For more than two weeks, Ford has been bombarded with questions about reports of an alleged video showing him smoking what appears to be crack cocaine. The mayor has denied using the drug.
Protester Jim Hedger arrived early for the rally and said the explosive video allegations are a tipping point for the city and Ford, who has been dogged by a string of controversies for much of his mayoralty, and before that as a city councillor.
"He's a man who appears willing to say anything — except the truth — to stay in office," Hedger said.
"He's not been good economically or culturally for this town and this is the final straw. He has to go."
The Facebook-organized protest — the first mass rally against the mayor since the drug video allegations broke — was peaceful and festive, with dogs, bikes and even a few babies dotting Nathan Phillips Square.
Though there were no speakers addressing the crowd, the rally found its voice with repeated chants of "hey hey, ho ho, Rob Ford's got to go" that bounced off the square's concrete slabs — which became increasingly chalked up with anti-Ford slogans during the protest.
"Ford resignation not Ford Nation," one white-chalk scrawl said, in reference to Ford's group of die-hard supporters.
"He's not our mayor — he's a joke," said Ford foe Brandon Fairley.
"We're a world-class city and in the past month people have been laughing at us. We've become a joke."
One backer of the mayor who braved the crowd wound up in a heated argument, but was quickly drowned out by the throng of Ford opponents.
Derek Hill said the protesters are jumping to conclusions about the purported video, which has not been publicly released.
"This is not due process. No matter how you feel about Ford, it's un-Canadian," he said.
Ford has insisted he is not going anywhere and will run again for mayor in next year's election.
Ford has refused to answer questions about the video scandal in recent days, brushing off reporters' queries with a terse "anything else." Ford spent a Friday news conference discussing an earlier shake-up of city public housing.
Several of Ford's staffers — including his chief of staff — have parted company with the mayor since the scandal erupted.
Also on HuffPost