The former foreign affairs minister replaces Marc Lortie, who has served as ambassador since 2007.
Cannon was first elected to the House of Commons in 2006 and re-elected in 2008, when he was named foreign affairs minister.
He lost his Quebec seat to a New Democrat in the 2011 election.
Cannon served in the Quebec National Assembly as a Liberal from 1985 to 1994.
The new ambassador will take office after presenting his credentials to president-elect Francois Hollande, who is to be inaugurated next Tuesday.
Harper said Cannon is an ideal choice for the job.
"His experience in the private and public sectors, as well as his significant diplomatic skill, will help ensure that Canada-EU trade negotiations are brought to a successful conclusion," the prime minister said in a statement.
He said Cannon's appointment "speaks to the importance of Canada’s relationship with France with whom we have important cultural, linguistic and historical ties."
Also on HuffPost