The 39-year-old Nestor has taken his usual place as the last Canadian in contention at a major after rising star Milos Raonic went out to Lleyton Hewitt in the third round of singles.
He and Mirnyi will next play either Wimbledon and US Open champions Jurgen Melzer of Austria and German Philipp Kohlschreiber, or the Mexican-German pair of Santiago Gonzalez and Christopher Kas.
Nestor and Mirnyi had to fight to hold onto a quick lead in the opening set and needed more than 90 minutes to finally achieve victory against the team whose only title came on clay in Bucharest last autumn.
"They were definitely tough, all the teams we have played so far have been tough," said Nestor. "They stepped up their game when we were leading in the first set and put pressure on us.
"Sometimes teams like that will then go away but they didn't; they and hung in there. It was the same in the second set when we were up a break.
Nestor and his partner have won four titles in 2011, including the London year-end finals and the Brisbane champions earlier this month.
"Max and I are pretty pleased with how we've been playing," added Nestor. "We' ve carried last year's momentum into the new year. Our goal is to win the Australian Open, we're trying to win the title here."
Nestor and Mirnyi led 5-2 before the Italian pair mounted a comeback, saving a pair of set points in the ninth game as Nestor was broken and two more in the 12th. With the set going into a tiebreaker, the drama continued.
Nestor's team got away to a 4-1 lead only to see the challengers push back, with Bracciale and Starace saving another pair of set points before Mirnyi finally produced a service winner on opportunity No. 6 to claim the opener after a hard-fought 57 minutes.
The second set was slightly less demanding as Nestor and Mirnyi started with a break and got another for 5-2 before serving out the win a game later.