Rae told his Liberal colleagues of his decision during Wednesday's party caucus meeting, an announcement that was met with tears and applause, according to sources.
Rae recently accepted the role of chief negotiator for First Nations in talks with the Ontario government about development of the Ring of Fire, and is to tour the nine Matawa communities in the mining and resource-rich area of Northern Ontario this summer. Rae is a former NDP premier of the province.
At a news conference with Liberal Leader Justin Trudeau in the foyer of the House of Commons, Rae said that work as a lawyer and mediator was taking more and more of his time and he felt he needed to focus on his role with the First Nations. He didn't say when his resignation would be effective.
"This is not about money, this is about time. I'm not being paid for my work in the North," Rae said, adding that his decision was about, "How can I effectively serve the interests of the people I've been asked to serve?
Rae thanked his caucus colleagues and said "above all, I want to thank my leader," saying he has enjoyed his time working with Trudeau. He added that he and Trudeau had been discussing his departure.
"It's been a very difficult decision and as you may have heard from the caucus, a very emotional one for me," he said, his voice shaking at times.
"It's not easy, frankly. The leader and I talked about this, and he expressed a strong desire for me to stay and I eventually had to tell him, sorry I can't do that."
He continued, "I'm quite confident that Mr. Trudeau will become the prime minister of Canada," Rae said.
Trudeau said he was saddened by Rae's decision, and that he will miss Rae's "wise counsel."
"We will miss his passion," Trudeau said.
"I am not letting him go far, [Rae] will continue be a valued friend, ally and resource for all Liberals in this country as we continue to build a brighter future for all Canadians," he added.
Rae recalled his days as a young NDP MP opposite Trudeau's father, former prime minister Pierre Trudeau. He said the rhetoric could get heated then but was not as calculated as the attacks he has seen recently in Parliament.
"We are in a particularly tough period but we'll get through it," Rae said, adding he hopes that over the summer the parties will realize that it can't go on this way.
"I think the public hates this stuff," he said, referring to dialogue in the House of Commons that has become nastier and also what he described as "more rote."
"What we see now is about as spontaneous as the Normandy invasion. I mean, every step that's taken, every question that's answered is answered with an attack. It's like a haiku form, you have to answer the question, you have to do this and then you have to attack."
He lauded Trudeau for remaining resilient in the face of personal attacks.
Asked if this decision meant closing the door on politics for good, Rae replied, "Look, never say never. But it certainly closes the door for now. But another door has opened and it's a big job."
But he did say the word "never" could be applied to being a candidate for mayor of Toronto, explaining not going after that job was a condition of his marriage.
Many Conservatives were among those reacting to the news of Rae's decision, including Prime Minister Stephen Harper and Heritage Minister James Moore.
"Wishing Bob all the best as he begins the next chapter of his life. One [of] Parliament's most passionate & effective voices," Moore tweeted.
Defence Minister Peter MacKay told reporters he considers Rae one of the most eloquent and capable speakers he's ever seen in the House of Commons.
"Bob Rae will be remembered fondly," MacKay continued, "as somebody who made a tremendous effort to contribute to public life. I think that's something that young people need today — positive role models — and Bob Rae would be considered one of those."
Speaking with reporters after NDP caucus Wednesday, NDP Leader Thomas Mulcair said, "This is a tough life in politics and on a day like this, all of us, whatever political stripe, simply have good thoughts for someone who has served for 35 years like Bob Rae and I wish him well."
NDP MP, Ontario premier
Rae has left politics before, but this leave-taking may well be the last one. His political career has spanned generations, and crossed party lines and provincial and federal jurisdictions.
He first arrived in Ottawa as an MP for the federal NDP in 1978 and stayed for four years. He left Parliament Hill to head an ailing Ontario NDP, and in 1990 became the premier of the province on a surprise election victory that saw some people in Toronto dancing in the streets until late into the night.
The honeymoon with some of his supporters during his tenure as premier faded as unhappiness grew over his economic policies, particularly over what came to be known as "Rae days," in which civil servants were forced to take unpaid leave.
He was defeated in the next provincial election and after a long stint as a lawyer, negotiator and mediator, he joined the Liberal Party and ran in the 2006 Liberal leadership race. He came third, finishing behind his old college roommate Michael Ignatieff and the winner Stéphane Dion.
He won a 2008 byelection in the Toronto riding of Toronto Centre and then won again a few months later in the 2008 general election. He was set to run for the Liberal leadership again at the end of that year, after Dion resigned, but withdrew. Ignatieff then took over the reins of the party.
After Ignatieff resigned following the disastrous showing for the Liberals in the 2011 election, Rae became interim leader of the party. There were strong rumours he would run again for the leadership, but he announced he would not seek the post that Trudeau handily won in April.
On Wednesday, Trudeau said of Rae, "I know that going forward, the history books will also remember someone who's a great leader for the Liberal Party, during a very difficult time for the Liberal Party."