Tory appeared on CBC Radio's Metro Morning Monday, one day after Toronto celebrated the conclusion of the Games with a spectacular fireworks show and a Kanye West concert.
Despite early worries about cost overruns and traffic chaos, Toronto delivered what many are calling a gold-medal performance as the host city.
On Saturday Canadian Olympic Committee President Marcel Aubut said he would lead and advocate for a Toronto 2024 bid. Paris, Rome, Budapest, Boston and Hamburg, Germany, are all said to be submitting bids.
However, Tory said while he's elated at how Toronto performed, hours after the Pan Ams wrap up isn't the time to talk about an Olympic bid.
"I just think today is not the day to answer that question," said Tory when asked by host Matt Galloway if an Olympic bid makes sense. "We should let things subside. We've got to make sure we execute the Parapan Games as well as we did the Pan Am Games."
The Parapan Am Games run Aug. 7 to 15.
Games have left a legacy
Tory spoke of the legacy of hosting the Pan Am Games, everything from the positive impression it made on 250,000 visitors and the benefits to Toronto that will last beyond the games.
He pointed to the Union Pearson Express train — which provides a direct link between downtown and Pearson airport — as a key piece of infrastructure that came with the Games.
"I believe that train would not be running today if we didn't have the Pan Am Games," said Tory.
Despite the benefits of hosting high-profile international events, Tory said the city must weigh the net benefit against the considerable cost of hosting an Olympics.
If Toronto is interested in making a bid, it will have to signal its intention to the International Olympic Committee by Sept. 15 in the form of a letter signed by Tory.
"There's a lot of consultation to be done with a lot of people as to whether they think it will advance the city's best interests. I want to do it … rationally and carefully so I'm not stampeded into anything. We're going to consider this carefully."