The Toronto native and partner Kristina Mladenovic of France lost in Thursday's mixed doubles final to Frantisek Cermak and Lucie Hradecka of the Czech Republic 1-6, 6-4, 10-6.
Nestor and Mladenovic, together for the first time, raced through the first set, the ninth straight set they had won at the tournament. But the Czechs broke Nestor in the opening game of the second, and rallied from 0-40 while serving for the set to even the match on a backhand volley from Hradecka.
"I'm a little bit disappointed, but I thought they played pretty well at the end," Nestor said. "It wasn't us. It was more them playing pretty well."
Nestor had a disappointing performance in men's doubles as well earlier in the tournament, losing in the second round with new partner Robert Lindstedt of Sweden — Nestor's worst result at Roland Garros since he was ousted in the second round in 2006.
Nestor had claimed the French Open doubles title the previous three years.
In Thursday's super tiebreaker, the Cermak and Hradecka won four straight points to lead 9-5, and clinched victory in the 69-minute match when Mladenovic netted a backhand volley, becoming the first Czechs to win the mixed doubles since 1991, when Helena Sukova and Cyril Suk triumphed.
"They were good in the tiebreaker," Nestor said. "I'm upset that I lost serve first game in the second because that kind of changed the momentum although we did have some chances to break back. For the most part, even on the points where we had chances they played pretty well."
Mladenovic, who at 20 is half Nestor's age, won the junior girls title here in 2009. Nestor, who has 80 doubles titles including eight Grand Slams, pledged afterwards to play with Mladenovic at Wimbledon.
Nestor will transfer onto the grass next week, playing with doubles partner Lindstedt in a pre-Wimbledon grass double — Queen's Club in London, followed by a first appearance on the south coast at Eastbourne at the joint ATP-WTA event.
It was the first Grand Slam title for Cermak and the second for Hradecka, who also won the women's doubles at Roland Garros with Andrea Hlavackova in 2011. Hradecka still has a shot in the women's doubles as she and Hlavackova will play their semifinal on Friday.
"I'm happy that we did it," Hradecka said. "It's really nice to be here in Roland Garros and stay on court on Chatrier and have a title."
Their very first match, at 2008 Wimbledon, was a loss, and they endured a bumpy road, though they reached the final at the Australian Open in January.
"We didn't start well," Cermak said with a laugh. "We're improving."
— With files from The Associated Press.