The mayor said he instead wanted a Canadian flag raised but it appears he will not get his way.
"This is about the Olympics, this is about being patriotic to your country," he said. "This is not about somebody's sexual preference."
When told the gesture was meant to protest anti-gay laws in Russia, Ford replied: "Let Russia do what they want. We're Canadians here.''
The Toronto Star's Daniel Dale tweeted that deputy mayor Norm Kelly was at the flag-raising ceremony.
Mayor Rob Ford is fighting for the removal of the rainbow flag Deputy Mayor Norm Kelly just helped raise over city hall for the Olympics.— Daniel Dale (@ddale8) February 7, 2014
Ford responded by putting up a Canadian flag in his office window.
The Pride flag was actually put up on a "courtesy" flag pole which otherwise flies the City of Toronto flag. Multiple flag poles around city hall fly the Canadian flag and the city flags at all times.
A city spokeswoman said the mayor didn't officially order the rainbow flag to come down, but did ask how it came to be raised.
"The mayor did ask to seek clarity on how this rainbow flag could be flown at city hall," said Jackie DeSouza, explaining that a non-profit group had asked to have the rainbow flag flown for the duration of the Sochi Games.
DeSouza said the group's request met the requirements of the city's flag flying policy, which was highlighted for the mayor.
"Given that city council approved this policy it would have to be up to council as a whole to change the policy," she said when asked if the mayor could order the flag to come down.
The flag flap drew some LGBT activists to the mayor's office to protest his comments.
"That he actually would go out of his way to remove a flag which is a symbolic gesture in support of those very basic human rights is probably more upsetting to me than anything he's done around here,'' Suzy Richter said after a conversation with the mayor.
Councillor Sarah Doucette told CP24's Katie Simpson she doesn't agree with Ford's position as there are already Canadian flags flying at city hall.
"If he's not prepared to change then I don't think he should be mayor," she said.
But Councillor Denzil Minnan-Wong said an Olympic or Canadian flag should be raised instead.
"This should be about sport and about Canadians," he said.
The comments come one day after Ford admitted that he does not attend gay Pride events because of personal preference, rather than an annual trip to his family's cottage.
Ford was asked at a debate on Wednesday if he would attend the World Pride festival in Toronto this June.
"I'm not going to go to the Pride parade," he said. "I've never gone to a Pride parade. So I'm not going to change the way I am."
Meanwhile, Ottawa Mayor Jim Watson responded with pure class on Twitter after he received a complaint about Ottawa also raising a rainbow flag, telling a man who complained about the move that he doesn't want his vote.
Many Canadian cities are flying the rainbow flag to protest Russia’s oppressive anti-gay laws.
With files from The Canadian Press
Also on HuffPost