Mayor Michael Nutter, who confirmed the deaths, said the scene was horrific and not all the more than 240 people on the train had been accounted for.
"It is an absolute disastrous mess," he said. "I've never seen anything like this in my life."
He said all seven train cars, including the engine, were in "various stages of disarray." He said there were cars "completely overturned, on their side, ripped apart."
"It is a devastating scene down there," he said. "We walked the entire length of the train area, and the engine completely separated from the rest of the train, and one of the cars is perpendicular to the rest of the cars. It's unbelievable."
More than 140 people went to hospitals to be evaluated or treated, and six were critically injured.
The cause of the derailment, which shut down service on the busy Northeast Corridor, was unknown. Amtrak said it was investigating and was trucking in lights to illuminate the wreckage overnight as workers examined it. The mayor, citing the mangled train tracks and downed wires, said, "There's no circumstance under which there would be any Amtrak service this week through Philadelphia."
Train 188, a Northeast Regional, had left Washington, D.C. The front of the train was going into a turn when it started to shake before coming to a sudden stop.
An Associated Press manager, Paul Cheung, was on the train and said he was watching Netflix when "the train started to decelerate, like someone had slammed the brake."
"Then suddenly you could see everything starting to shake," he said. "You could see people's stuff flying over me."
Cheung said another passenger urged him to escape from the back of his car, which he did. He said he saw passengers trying to escape through the windows of cars tipped on their sides.
"The front of the train is really mangled," he said. "It's a complete wreck. The whole thing is like a pile of metal."
Another passenger, Daniel Wetrin, was among more than a dozen people taken to a nearby elementary school afterward.
"I think the fact that I walked off (the train) kind of made it even more surreal because a lot of people didn't walk off," he said. "I walked off as if, like, I was in a movie. There were people standing around, people with bloody faces. There were people, chairs, tables mangled about in the compartment ... power cables all buckled down as you stepped off the train."
The area where the derailment occurred is called Frankford Junction and has a big curve. It's near where one of the nation's deadliest train accidents occurred: the 1943 derailment of The Congressional Limited, from Washington to New York, which killed 79 people.
Police swarming around Tuesday's derailment site, in the working-class Port Richmond neighbourhood, told people to get away from the train. They pleaded with onlookers: "Do NOT go to scene of derailment. Please allow 1st responders room to work."
Roads around the derailment site were blocked off. Waves of firefighters surrounded the train cars, taking people out.
Several injured people, including a man complaining of neck pain, were rolled away on stretchers. Others wobbled while walking away or were put on buses. An elderly woman was given oxygen.
Former Congressman Patrick Murphy was on the train and said he helped people. He tweeted photos of firefighters helping others in the wreckage.
"Pray for those injured," he said.
Amtrak said the train was carrying 238 passengers and five crew members.
The National Transportation Safety Board said it was gathering information about the derailment and would send an investigative team to the site Wednesday morning. The Federal Railroad Administration said it was dispatching at least eight investigators to the scene.
Port Richmond is one of five neighbourhoods in what's known as Philadelphia's River Wards, dense rowhouse neighbourhoods located off the Delaware River. Area resident David Hernandez, whose home is close to the tracks, heard the derailment.
"It sounded like a bunch of shopping carts crashing into each other," he said.
The crunching sound lasted a few seconds, he said, and then there was chaos and screaming.
Gov. Tom Wolf, who surveyed the damage with the mayor, thanked the first responders for "their brave and quick action."
"My thoughts and prayers are with all of those impacted by tonight's train derailment," he said. "For those who lost their lives, those who were injured, and the families of all involved, this situation is devastating."