People living south of Ninth Avenue between 13th and 15th streets were told to leave.
"Small evacuation of immediate area complete, situation coming under control. Stay tuned," tweeted Calgary Mayor Naheed Nenshi.
Canadian Pacific Railway says eight tanker cars containing diluent came off the tracks as a train was leaving the Alyth Rail Yard located at 19th Avenue and 15th Street.
A CP spokesperson says they're not aware if any of the cars are leaking but nobody was injured.
Petroleum diluent is used to thin petroleum products, including bitumen from the oilsands, to help transport them through pipelines.
CP has a response team at the scene. Police are advising traffic to avoid the area.
Officials are planning to close some roads, including parts of Blackfoot Trail and Ninth Avenue.
Some Calgary Transit routes near the scene are also being detoured.