The move is expected to affect 350 workers.
"The water heater industry is extremely competitive and has been burdened with overcapacity for a number of years," said Kevin Wheeler, president and general manager of North American water heating in a release.
"In addition, we have been facing a challenging economic environment as a Canadian manufacturer."
The company said it expects to end the majority of production at the Fergus, Ont., plant on July 1.
"This was a tremendously difficult decision to make, given the long history of the Fergus operation and its experienced workforce," he said.
"However, after careful consideration of all of the alternatives, we decided it was necessary to cease manufacturing in Fergus. We understand that this will create a great deal of uncertainty with our Fergus employees, and we are committed to helping them during this difficult transition."
Fergus will remain the head office of the company's Canadian business with a total of about 125 employees.
An A.O. Smith plant in Stratford, Ont., which manufactures pump tanks for water systems will not be affected.
The factory in Fergus manufactures residential and light commercial water heaters for Canada and the U.S.
The company said it will meet with employees this week to discuss the closure.
Affected employees will be offered severance benefits as well as job placement assistance.
A.O. Smith, based in Milwaukee, Wis., employs 10,900 people in the United States, Canada, Mexico, China, the Netherlands, and India.
It sells water heaters under the A.O. Smith, GSW, and John Wood brands in Canada.