While his regular rivals Roger Federer and Novak Djokovic won through to the quarterfinals, Murray lost 6-4, 6-4 to Chardy, an opponent he'd beaten easily the four previous times they'd played.
Murray had trouble controlling his shots on a breezy, hot afternoon.
"I was a little bit uncomfortable with those balls bouncing extremely high," said Murray. "It was fairly hard to control."
Murray won a gold medal in the Olympics at Wimbledon by beating Federer, the highlight of his career. He went to Toronto last week hoping to get in shape for the hard-court season, but had to withdraw from the Rogers Cup because of a minor injury to his left knee.
Murray felt good and moved well in an opening two-set win on centre court Wednesday. Playing on the grandstand court a day later, the Scot never got into a rhythm.
Now, Murray heads to New York to practice for the U.S. Open, having played only three matches on hard courts since returning from Wimbledon's grass.
"I won't play any more matches," Murray said. "When going into the big tournaments, sometimes I've won tournaments in the buildup and it hasn't helped me. This year at Wimbledon, I lost in the first round at Queen's and made the final there for the first time. I have won Queen's before and not done well at Wimbledon.
"So it doesn't normally have that much bearing. But I would have liked to have done a bit better this week."
Chardy got into the tournament as a replacement for John Isner, who had a sore back. The Frenchman won his opener over Andy Roddick, who developed back spasms during the match on Tuesday.
Chardy is the first injury replacement to reach an ATP Tour quarterfinal this season.
Federer beat Bernard Tomic 6-2, 6-4 and has looked sharp in his two hard-court matches. Federer skipped the Rogers Cup last week.
"I feel like things are working really well for me," Federer said. "Regardless of what's going to happen, I'm feeling good, I'm healthy, I have no issues. I'm happy I got over the jet lag, and so the risk is less to get injured and all those things. So I'm in a good place right now."
Djokovic was on the court for only 30 minutes in the evening. Russia's Nikolay Davydenko quit after losing the first set 6-0, bothered by a sore right shoulder. He called the trainer over at the break, then decided not to continue.
Davydenko warmed up fine, but his serve quickly deteriorated. He had eight double faults, made 16 unforced errors and won only 16 points.
"You definitely don't want to win like this," Djokovic said. "He couldn't serve over 90 mph (by the end of the set) and he had a lot of double faults. Something was bothering his shoulder. Hopefully he'll be able to recover in time for the U.S. Open."
On the women's side, top-seeded Agnieszka Radwanska overcame an unexpected challenge from wild-card Sloane Stephens before winning 6-1, 4-6, 6-4. Radwanska, who lost to Serena Williams in the Wimbledon final, reached the tournament's quarterfinal for the first time in three appearances.
Like Murray, she had trouble with the condition on the grandstand court.
"It was so hot and windy," she said. "I think that this court was slower than the centre (court) and the ball didn't go at all. I was hitting as much as I could, but it just was hard to make winners there."
Serena Williams won her 19th straight match — her last 12 in straight sets — by beating Urszula Radwanska of Poland 6-4, 6-3. Williams hasn't lost a set since the Wimbledon final.
She returned to Wimbledon and won the gold medal in singles and in doubles with sister Venus. The travel and the hot conditions have worn her down.
"I have been playing a lot," she said. "Definitely (need) more energy. Been travelling a lot, been training a lot. You know, it's been a really big summer."
Venus Williams beat Sara Errani 6-3, 6-0, a quick match after two long ones over the previous two days.
"I already got what I wanted at the Olympics this year, so everything else for me is just amazing," Venus Williams said.
Mardy Fish, a two-time runner-up at the tournament in suburban Cincinnati, beat Radek Stepanek 6-3, 6-3, his first win over the Czech in five career matches.
"That was the monkey jumping off my back," Fish said. "He's been a tough opponent for me. Not only has he beaten me, but he's beaten me soundly."
Fish continues to improve after undergoing surgery in May to repair a heart condition.
"I feel fresher, stronger," he said. "The summer's gone well for me as far as getting my match fitness back."
AP freelancer Mark Schmetzer contributed to this report.Suggest a correction