Nestor and Zimonjic, the tournament's top seeds, lost 6-4, 6-3 to Rohan Bopanna of India and Pakistan's Asiam-Ul-Haq Qureshi.
Nestor and Zimonjic beat Bopanna and Ul-Haq Qureshi in the doubles final of a tournament in Sydney last month. But they were outplayed this time by Bopanna and Ul-Haq Qureshi, who needed 68 minutes to secure the win.
"They were better, we probably didn't get enough first serves in," said Nestor. "When we both had chances to play they put more pressure on us.
"We just didn't play well all around."
Nestor, 41, was playing in his third ATP final in five weeks, winning in Brisbane and Sydney while also earning the Australian Open mixed title with Kristina Mladenovic of France.
Nestor won the Dubai title in 2002 with Mark Knowles of the Bahamas but was unable to repeat the feat 12 years later.
He and Zimonjic saved three set points in the final game of the 35-minute first set. But they were finally caught by a shovelled winner down the line which ended a fierce volley duel at the net between all of the four players.
Bopanna and Ul-Haq Qureshi took control of the second set with a break of Zimonjic for a 3-1 lead en route to the victory.
"We've won some matches this week so that's a good thing," said Nestor. "Doubles seems to turn on just a few points, who knows anything can happen.
"We can beat anyone or lose to anyone at any time."
Nestor still needs a win to move into third all-time in the post-1968 Open. He currently has 83 career wins, tied with Australian Todd Woodbridge and behind Mike Bryan (96) and Bob Bryan (94).
Nestor and Zimonjic have won 25 titles together with Nestor having played in 137 doubles finals.