A victory by the unseeded Canadians would have ended the Americans' bid for a 2013 Grand Slam sweep. The brothers won earlier this year at the Australian Open, French Open and Wimbledon.
"We weren't thinking of that at all," said Nestor. "We were just trying to win: big match, centre court, TV, you want to win those matches.
"But to beat the Bryans you have to play your best, Until a set and 4-3 we were playing as well as we could. But I didn't continue it with my serving."
Nestor, from Toronto, was broken over his last four service games and also set up match point for the Bryans with his team's eighth double-fault. Nestor's serving troubles began midway through the second set and the Americans took the momentum in a match that lasted just over two hours.
"I didn't serve well from 4-3," said Nestor. "It was unfortunate, it would have been nice to go into a tiebreaker in the second set.
"I could have mixed up my serve a bit more, I was getting repetitive with the first serve, which was not going in. My serve was definitely the difference today."
With Nestor's game off the mark and Pospisil, from Vernon, B.C., short of big-match experience, the Bryans were able to claw back in front of a supportive crowd. The Canadians managed to break on only one of seven chances despite producing well over 50 winners with their attacking style.
The brothers pulled even at a set apiece after another break of Nestor, who remains alive in the mixed event with partner Kristina Mladenovic of France. Nestor and Mladenovic won at the French Open in June.
Nestor has faced the Bryan brothers 53 times over his career with seven different partners.
"It was one of those matches where we were the better team until the end of the second set," Pospisil said. "We got a bit unlucky on the outcome of a few challenges and the match turned. It was a few important points in crucial moments."