Nestor and Zimonjic entered the tournament on a roll after winning back-to-back clay titles at Madrid and Rome. However, Draganja and Mergea took the third-set tiebreaker after trailing 4-1 to upset the third-seeded duo.
"It's disappointing to lose but especially when you feel you have a chance to win a Slam," said Nestor. "We were prepared for this match. We had also gotten out of danger in previous matches this season and dealt with situations well.
"It just didn't work out today."
The match lasted two hours 18 minutes. Nestor had been bidding for a seventh Paris semifinal appearance.
After Draganja and Mergea tied the match at a set apiece, Zimonjic was broken to start the third. They got the break back to move to 4-4 but were unable to prevent the tiebreaker.
The underdogs earned the victory on their first match point to conclude a final set which took just over an hour.
"Nobody is a nobody," Nestor said of the relatively unknown winning side. "They returned well, with like 20 clean winners. It's unfortunate we didn't do something with our chances in the tiebreaker. Every match is close from the first round.
"We didn't do enough with some returns and should have played better."
The 41-year-old Canadian is still alive in the mixed doubles competition with Kristina Mladenovic of France. Nestor and Zimonjic had won their last 11 clay matches and finished the clay campaign with a 16-3 record.