A spokesman for the talks says Ontario Chief Justice Warren Winkler, who was leading the proceedings, has concluded that further efforts at mediation were no longer worthwhile.
The negotiations began in Toronto on Jan. 14 and were supposed conclude that week. They were extended twice.
This was the third attempt at settling claims against the former telecom manufacturing giant's business lines and intellectual property by about 100 creditors around the world, including companies in 20 countries on every continent except Antarctica.
The myriad of creditors include Nortel pensioners, disabled former employees, bond holders, trade creditors and governments.
It was not immediately clear what next steps would be taken.
When he was appointed to the role by courts in the U.S. and Canada, Winkler admitted the complex negotiations would be difficult to resolve because the value of the claims exceeded Nortel's residual assets.
At its height, the Ottawa-based equipment provider was worth nearly $300 billion and employed more than 90,000 people worldwide.
During the technology boom in 1999-2000, Nortel was considered one of Canada's most valuable companies, with its shares peaking at $124.50.
In the years that followed amid an accounting scandal with its top executives, the company's shares nosedived to penny-stock status.
In 2009, Nortel filed for bankruptcy in North America and Europe, shedding thousands of jobs.
Since then, it has sold its remaining businesses piecemeal to various buyers for more than to US$7.8 billion, one of largest asset sales in Canadian history.
Last week, the former top brass at Nortel were acquitted of fraud charges nearly a decade after being accused of falsifying financial records at the beleaguered company.
Ex-CEO Frank Dunn, ex-CFO Douglas Beatty and ex-controller Michael Gollogly had been fired in 2004 and accused of being involved in a book-cooking scheme to trigger $12.8 million in bonuses and stock payments to themselves.
But a judge found there was no evidence to support the charges of fraud against the executives.