The 40-year-old Nestor, who has won the most doubles titles in history, was unable to nose his fifth-seeded team into the quarter-finals.
"They put a lot of pressure on us and conditions favoured them a bit more," said Nestor. "Every day feels different here. It depends on what court you are on.
"This one felt slower than where we had practised. Conditions were also heavier with the balls and the air."
The third-round defeat in just under two hours was a disappointment for Nestor, who split with Max Mirnyi of Belarus last season to team up with India's Bhupathi.
Monday's match was marred by eight double-faults from Nestor and Bhupathi, who also converted on just three of ten break chances.
Italy's Boelli and Fognini won the opening set in 33 minutes through a break in the seventh game. But Nestor and Bhupathi redeemed themselves in the second, squaring the match at a set each as they broke for 4-3 on the way to winning 6-4.
But in the third set, Nestor was broken in the fourth game on a double-fault to trail 3-1. Nestor and Bhupathi hung tough but wound up falling 6-3 to crash out of the tournament.
Nestor said that a mix up in communication and a moment of self doubt led to the double-fault which cost him the fourth game of the final set.
"There was some confusion on the last break point. I was indecisive on what to do and I relied on advice from my partner instead of going with my gut feeling," he said. "As a server, you want to do what you think is best. It stings a little bit but it was my fault.
"I made the wrong play due to indecision."
Milos Raonic of Thornhill, Ont., was set to take on No. 2 seed and four-time champion Roger Federer in a fourth-round singles match later Monday.