The city hall flag flap, which happened while the opening ceremonies were underway in Russia Friday, began after the flag was raised in solidarity with other cities also flying the flag to show support for gay rights.
Ford responded by putting a Canadian flag in his office window.
"This is about Olympics," he told reporters. "This is about being patriotic to your country, this is not about someone's sexual preference."
Deputy Mayor Norm Kelly said he supports keeping the rainbow flag up, saying it sends a message to the host country, which has been under fire for its anti-gay laws.
"There's no antagonism between the two flags," said Kelly. "Cities right across the country are doing this. This is an expression of Canadianism."
Kelly said he hoped Ford would "reconsider his position" and that the mayor should "keep the bigger picture in mind."