Flaherty's family released a statement that said he died Thursday in Ottawa. No cause was made public.
Flaherty, who had the job since 2006, was the longest-serving finance minister among the Group of Seven leading industrial economies until he announced he was stepping down three weeks ago to return to the private sector. He battled a rare skin disease over the last year, but had said his decision to leave politics was not related.
Flaherty was Prime Minister Stephen Harper's only finance minister since Harper took power eight years ago. He is credited with helping get Canada back on track to a balanced budget after pumping stimulus money into the economy following the 2008 financial crisis.
The House of Commons in Ottawa abruptly suspended business as word of his death circulated. Harper addressed members of his government in brief comments that were televised across the nation
"I learned a short while ago that our colleague, my partner and my friend, Jim Flaherty, has passed away suddenly today," said Harper. "This comes as an unexpected and a terrible shock ..."
Opposition New Democrat leader Thomas Mulcair choked back tears as he expressed condolences to the family.
"We're very, very sad at the loss of a great Canadian. Jim Flaherty was an extraordinarily dedicated public servant," Mulcair said.
The flag on the Peace Tower on Parliament Hill in Ottawa was lowered to half-mast.
Ottawa police said they responded to a medical call shortly after noon at Flaherty's residence. He is survived by his wife Christine Elliott and his triplet sons, John, Galen and Quinn.
Born in Lachine, Quebec, Flaherty went on to earn a bachelor's degree from Princeton University and a law degree from York University's Osgoode Hall Law School.
Flaherty spent almost 20 years in politics, including stints as minister of finance, attorney general and deputy premier at the provincial level in Ontario. In 2006, after two unsuccessful bids to lead the Ontario Conservatives, Flaherty entered federal politics.
Harper's Conservative government plans on entering an election next year with a budget surplus, and the prime minister has praised Flaherty for helping make that happen.
Canada's commodity-rich economy avoided the worst of the crisis and has fared better than other nations. There was no mortgage meltdown or subprime crisis in Canada.
Flaherty took over from former Liberal Prime Minister Paul Martin's Liberals in 2006 and broadly stuck to his predecessor's approach, though Flaherty cut taxes and, when recession struck, pumped stimulus money into the economy. He was also an outspoken critic of European countries for their handling of the debt crisis and urged them to get their debt problems under control.
Flaherty acknowledged last year that he was suffering from a skin condition, requiring him to take medication that led to weight gain. He also appeared fatigued at some public appearances.