With all polls reporting, the Parti Québécois's Cardin had 42 per cent of the vote to Charest's 35 per cent.
Coalition Avenir Québec candidate Philippe Girard trailed with 12 per cent.
Charest conceded defeat in a speech at about 11:20 p.m. ET. He congratulated Cardin and thanked Sherbrooke voters, whom he has represented at the federal and then provincial level for 28 years.
"I'm very proud to be a Quebecer, and I'm also very proud to be a Canadian. And I want to say to all of you tonight, and all of you interested in the future of Quebec, that the result of this election campaign speaks to the fact that the future of Quebec lies within Canada," Charest said in his speech.
Charest has come close to losing his provincial seat before. He won by a margin of 3.7 per cent in the 2007 Quebec election, and held on by fewer than eight points in 2003 and 2008.
He becomes the first sitting premier since Robert Bourassa in 1976 to lose his own riding.
Cardin had strong support in Sherbrooke during the campaign. Charest's own former justice minister Marc Bellemare, among others, came out in support of the PQ candidate.
Charest first won the provincial seat in 1998, after he resigned as federal leader of the Progressive Conservative Party to take the helm of the Quebec Liberals. In the byelection to replace him in the House of Commons, Cardin won the riding for the Bloc Québécois.