They gathered across the street from the U.S. Consulate, a day after a grand jury in Missouri decided not to indict Ferguson police officer Darren Wilson in the fatal shooting of Brown, who was 18.
Protest leader Lena Peters said the immediate issue was that the grand jury made the decision not to take the "murder" of a black man to trial.
"Those folks who are here tonight don't believe that's OK," she said.
"We believe that it is part of a larger system that doesn't value black life."
Just a few police officers were present, reflecting the peaceful nature of the protest, which drew some 3,000 people.
Holding candles, they repeatedly chanted, "No justice, no peace," and "Being black is not a crime."
"Basically, I don't want to live in a world where some lives matter and some lives don't," said Christina Miniaci, one of the protesters.
The grand jury's decision sparked a riot in Missouri overnight and thousands of National Guardsmen were called in in an effort to prevent a second night of violence.
Lawyers for Brown's family have condemned the judicial process that led to the officer not being indicted as unfair and broken, and President Barack Obama has called for restraint in the wake of the ruling.
The Toronto protest leaders also called for an acknowledgment by all levels of the Canadian government of what they called the excessive use of police force in Ferguson against blacks as a human rights issue.
Peters and others also used the occasion to vent their anger against police in the Greater Toronto Area for alleged racial profiling and brutality against black people.
They demanded justice for Jermaine Carby, who was fatally shot by Peel Regional Police during a traffic stop in Brampton, Ont., in September. Carby's cousin, La Tanya Grant, was among the speakers at the event.
"As much as we are here for the Ferguson situation, let us try to support my campaign for Jermaine Carby to try, but at least get justice here in Toronto."
Ontario's police watchdog is investigating that case.
Meanwhile, a suggestion that white and non-black protesters limit their visibility and "stand behind black folks" at the vigils in Toronto and Ottawa sparked a heated debate on social media.
Bilan Arte, one of the Ottawa event organizers, said on the vigil's Facebook page that "white/non black allies" should "refrain from taking up space" and "never be the centre of anything."
The same message appeared verbatim on the Facebook page for the Toronto rally, asking whites and non-blacks to not speak to the media, saying "black voices are crucial to this."
Arte's post prompted more than 100 comments, with one man asking "is this an anti-racist rally or a pro-segregation one?"
Arte, deputy chairwoman with the Canadian Federation of Students, said on her own Facebook page that she received upwards of 50 messages from "righteous upset white people" taking issue with the post.
But there were many non-blacks among the protesters in the Toronto event, which was organized through social media by an online forum called Black Lives Matter.
Note to readers: This is a corrected story. An earlier version had an incorrect name for Jermaine Carby's cousin La Tanya Grant.