The flag — a long-standing symbol of lesbian, gay, bisexual and transgender rights and pride — was raised in a sign of solidarity, but Rob Ford said he wanted it taken down.
"This is about the Olympics. This is about being patriotic to your country. This is not about someone's sexual preference," he said.
When told the gesture was meant to protest anti-gay laws in Russia, Ford replied: "Let Russia do what they want. We're Canadians here."
The flag was put up on a "courtesy" flag pole which otherwise flies the City of Toronto flag. Multiple flag poles around city hall fly the Canadian flag and the city flags at all times.
A city spokeswoman said the mayor didn't officially order the rainbow flag to come down, but did ask how it came to be raised.
"The mayor did ask to seek clarity on how this rainbow flag could be flown at city hall," said Jackie DeSouza, explaining that a non-profit group had asked to have the rainbow flag flown for the duration of the Sochi Games.
DeSouza said the group's request met the requirements of the city's flag flying policy, which was highlighted for the mayor.
"Given that city council approved this policy it would have to be up to council as a whole to change the policy," she said when asked if the mayor could order the flag to come down.
The flag flap drew some LGBT activists to the mayor's office to protest his comments.
"That he actually would go out of his way to remove a flag which is a symbolic gesture in support of those very basic human rights is probably more upsetting to me than anything he's done around here," Suzy Richter said after a conversation with the mayor.
The scene at city hall momentarily intensified Friday when police were seen entering the mayor's office, prompting speculation that their visit might be related to the flag issue.
A spokesman for the mayor quickly clarified that wasn't the case, saying only that police were investigating a "potential threat" to Ford and his family.
Toronto was one of several Canadian cities — including Montreal, Ottawa and Edmonton — to fly the rainbow flag on Friday.
In Vancouver, the flag went up at the request of Mayor Gregor Robertson, and coincided with a decision to send an openly gay councillor to Sochi to lobby for the International Olympic Committee to make Pride Houses mandatory at future Games.
In Montreal, Mayor Denis Coderre said his city protested Russia's anti-gay laws.
"Over the past months we have observed several gestures of repression toward the LGBT community in Russia," Coderre, a former federal sports minister, said as the rainbow flag was raised.
"This gesture is a symbol of fraternity and solidarity toward the LGBT community of Montreal and around the world...Today, we are all Russian for all those people who are suffering right now."
Ford, who lost most of his mayoral powers late last year after admitting he'd smoked crack cocaine while in office, has previously come under fire for his staunch refusal to attend Toronto's gay pride parade that marches through the downtown core each summer.
In the past, he said his refusal to attend was due to a family tradition of spending the Canada Day long weekend at the cottage.
But when asked Wednesday if he was planning to attend this year's World Pride parade, Ford said: "I've never been to a pride parade. So I'm not going to change the way I am."
The mayor's remarks prompted criticism from the public and at least one fellow city councillor, who accused him of "thinly veiled homophobia."
The mayor's councillor brother, Doug Ford, denied that either he or his brother are homophobic. He said he felt uncomfortable bringing his kids to an event where "middle-aged men with pot bellies" ran down the street "buck naked."
_ With files from Diana Mehta.