Socialist party leader and former finance minister Evangelos Venizelos, the third party leader this week to fail at the task, said he would hand the mandate back to the president on Saturday. The president will then bring all party leaders together in a last-ditch attempt to create a coalition.
Greece has plunged into political turmoil since elections on Sunday gave no party enough seats in Parliament to form a government. Voters furious at two years of harsh austerity measures taken in return for international bailouts worth €240 billion ($310 billion) rejected Greece's two formerly dominant parties, the socialist PASOK and the conservative New Democracy, in favour of a myriad of smaller parties on the left and right.
The anti-bailout Radical Left Coalition, or Syriza, led by Alexis Tsipras made the most gains, coming in second with 16.8 per cent of the vote and 52 seats in the 300-member parliament, campaigning on a pledge to overturn the austerity measures.
Tsipras has refused to join any coalition government that says it will implement the bailout deal.
"The rejection of this plan does not come from Syriza but was given by the Greek people on the night of the election," Tsipras said after no solution was reached in his Friday night meeting with Venizelos. "The bailout austerity has already been denounced by the Greek people with its vote, and no government has the right to enforce it."
The other political leaders insist his policy is irresponsible and will force Greece out of the euro, but also say his party is essential in any power-sharing deal after coming second in the election.
The political instability has alarmed Greece's European creditors and rocked the Athens stock exchange, which closed 4.52 per cent down Friday even before news of the failure to reach an agreement broke. The exchange has fallen every day this week except Thursday.
International creditors have warned that the country's international bailout loans and its membership in the 17-nation eurozone could be threatened. German Finance Minister Wolfgang Schaeuble even suggested the eurozone could deal with an abrupt Greek exit.
"We have learned a lot in the last two years and built in protective mechanisms," Schaeuble told the Rheinische Post newspaper. "The risk of effects on other countries in the eurozone have been reduced and the eurozone as a whole has become more resistant."
Venizelos said he had found common ground in talks with Fotis Kouvelis, head of a small left-wing party, and New Democracy head Antonis Samaras, who won the election with 18.9 per cent of the vote, for a broad coalition that would have a two-year mandate and seek to secure Greece's participation in the euro.
But hopes that would work were dealt a blow earlier Friday when Kouvelis said he could not agree to join in a partnership without the support of Tsipras.
Greek President Karolos Papoulias could still manage to break the deadlock, but chances appear slim.
"I hope that during the negotiations chaired by Mr. Papoulias everyone will be more mature and responsible in their thinking," Venizelos said.
Greece has been dependent since May 2010 on rescue loans from other European Union countries that use the euro and the International Monetary Fund. In return, Athens has imposed repeated rounds of spending cuts and tax hikes, leaving the country mired in a fifth year of recession with unemployment above 21 per cent.
Tsipras could be looking forward to a new election. One opinion poll showed that his party would likely come first in a new ballot, although with not enough votes to form a government by itself.
The Fitch ratings agency warned that the outcome of the coalition talks or a new election would be critical.
"The election or formation of a Greek government unwilling or unable to abide by the terms of the current EU-IMF program would increase the risk of Greece leaving the eurozone," Fitch said.
The agency said if Greece did leave the euro, it would likely place all 16 remaining euro nations' sovereign ratings on "rating watch negative" — indicating they were in danger of being downgraded.
"A Greek exit would break a fundamental tenet underpinning the euro — that membership of EMU is irrevocable," Fitch said.
EU monetary affairs chief Olli Rehn stressed that Greece's bailout terms were the only way the country could reform its economy.
"Greece systemically lived beyond its means for a decade. ... It is simply not sustainable and therefore Greece has had to take firm action to restore its economic competitiveness and sustainable public finances," he said in Brussels.
Derek Gatopoulos in Athens, Raf Casert in Brussels and David Rising in Berlin contributed.