They also urge city politicians to consider the social costs of allowing a casino development within Toronto's city limits.
The letter was penned by former mayors David Crombie, John Sewell and Art Eggleton.
"We saw countless proposals that proponents said would bring the city big revenues, many jobs and much prestige," the joint letter says.
"Beware the sales pitch. The numbers don't always add up."
Two big-time casino operators have already pitched their vision for gambling in downtown Toronto.
MGM has proposed building a casino and resort at Exhibition Place, while Caesar's has expressed its plan to build a gambling facility on the grounds of the Metro Toronto Convention Centre.
Both casino developers have promised revenue and jobs for the city, but the three former Toronto mayors claim any casino would bring its share of costs as well.
"The costs are reflected in infrastructure costs," the letter states, "higher regulatory expenses, additional policing and criminal justice costs and larger social welfare expenditures."
The letter also says a casino would bear social costs.
"It preys on the poor and those vulnerable to addiction," the three mayors wrote. "Studies indicate that where casinos go, so go increases in poverty, homelessness, gambling problems, suicides, drug/alcohol addictions, and crimes by those feeding their habit."
"Governments shouldn't be expanding gambling opportunities as a means of balancing their budgets," the joint letter concludes. "A commercial casino operation is not in Toronto's best interest."
The city of Toronto launched a series of public consultations on whether to go ahead with some form of casino development in November.
Toronto city council must give the province a yes or no answer on a downtown casino by the spring.