Cohen and Sullivan, who broke the Olympic-best time in the heats, produced a late surge to overtake Italy with around 200 metres remaining and win in 6 minutes, 31.67 seconds under threatening skies.
The Italians, racing in lane 6, were a half length back to take a surprise silver and Slovenia won the bronze, having led for much of the race.
It was New Zealand's first gold of the London Games.
The race was without the defending champions from Australia, David Crawshay and Scott Brennan, and also the favoured German crew of Hans Gruhne and Stephan Kruger. Both boats were eliminated in the semifinals, meaning the field was wide open for the first final of the day.
The Slovenian boat of former Olympic champions Luka Spik and Iztok Cop charged into an early lead and were two seconds clear of Argentina by the halfway mark, with a stroke rate as quick as anybody else's in the race.
Italy's Alessio Sartori and Romano Battisti, who only reached the London Olympics by winning the final qualification regatta, then moved past Slovenia with 550 metres to go but couldn't hold off the Kiwis.
Cohen and Sullivan won by 1.13 seconds from the Italians, with Spik and Cop 1.55 seconds further back. Argentina was fourth.