Prior to the confrontation, O'Toole, two other Conservative MPs and Fantino's chief of staff met with the group of ex-soldiers, but failed to ease the concerns of the veterans, who were upset over planned office closures.
As recounted by some who were in the room, the encounter is instructive for the veterans community, which is wondering whether O'Toole's fresh face will actually mean a fresh approach in a department that has become a political liability for the Conservatives in the run-up to this year's election.
In a quiet ceremony Monday at Rideau Hall, Prime Minister Stephen Harper replaced Fantino with O'Toole, a southern Ontario MP and former Sea King helicopter navigator who was elected to the House of Commons in a 2012 byelection.
Fantino, a tough-talking former police chief who represents the strategically important riding of Vaughan, north of Toronto, remains at the cabinet table as associate minister of defence.
But repairing the political damage of his 18-month tenure, which was marred by controversy, confrontation and cries of incompetence, will fall to O'Toole, who has been a frequent, articulate defender of the government's policy in both the House of Commons and in the media.
"It's an honour for me to serve our veterans," O'Toole said after the ceremony while getting into his car at Rideau Hall. "I am getting up to speed on what I need to know, but it's a privilege for me to serve our veterans."
A spokesman for the Royal Canadian Legion says the minister won't need extensive briefing, given that he is a veteran and has been active on the file — most notably as a organizer of a Parliament Hill event last spring meant to raise awareness of post-traumatic stress.
"We're all hoping nobody says, 'You've got to give him time to get up to speed'; not that guy, not Erin O'Toole," said Scott Ferris, the legion's marketing director.
The Harper government is expected to deliver by the end of January a more substantive response to last year's report from the Commons veterans committee, Ferris added.
"Erin O'Toole has been sitting in the wings for how long? He's a veteran. He knows."
Others, such as Mike Blais of Canadian Veterans Advocacy, were doubtful the shuffle would allow the government to rebuild its bridges with veterans, a political constituency important to the Conservatives.
What needs to happen is a fundamental shift in the culture of the department from an insurance company mentality to one of compassion and support, he said.
"Changing the messenger will only help if the message is changed," Blais said. "Appointing someone new with the same lame rhetoric is not effective change. It is damage control."
NDP Leader Tom Mulcair said O'Toole could make a fresh start by reopening the nine Veterans Affairs offices boarded up last year as part of the department's reorganization.
Liberal veterans critic Frank Valeriote expressed some sympathy for Fantino, describing him as a "good soldier" who had to deliver the same old "sour message" from the Prime Minister's Office.
Ron Clarke, a veteran who confronted Fantino before the cameras last year and has promised to campaign against the Conservatives, was less forgiving. The demotion makes no difference to him whatsoever, Clarke said.
After the ceremony, Fantino didn't speak to reporters other than to offer New Year's greetings. But in a written statement, he later defended both his own record and that of the government in their treatment of ex-soldiers.
"Each and every day that I served at Veteran Affairs I was guided by a firm belief that government must stand by those who have served and continue to serve," Fantino said.
"Under Prime Minister Harper, I can say with confidence that we have fully embraced that principle."
Fantino went on to list some of the recent improvements, announced in November, including the expansion of services to deal with operational stress injuries.
A former street cop who later became Toronto police chief and commissioner of the Ontario Provincial Police, Fantino entered federal politics by winning a byelection in 2010.
He was re-elected in 2011 and named as associate minister of defence before becoming minister of international co-operation in July 2012. From there he went to Veterans Affairs.
Fantino found himself in political hot water almost from the moment he took the post.
Ever since, the department has been under heavy fire from veterans groups, the veterans ombudsman, the auditor general and the political opposition.
Fantino's efforts to defend office closures, job cuts, lapsed budget money and tweaks to pensions and benefits provoked anger from veterans and scorn from the NDP and Liberals.
There were public relations gaffes, including a much-publicized — and televised — confrontation with veterans, and the spectacle of Fantino walking away from the wife of a former soldier, ignoring her shouted questions.
Then there was the government's decision to go to court to argue that it had no special responsibility to care for veterans, despite the fact that such a responsibility had been iron-clad policy since the First World War.
There were howls of protest when it was learned that since 2006, the department had allowed more than $1 billion of its budget to lapse and be returned to the federal treasury. The anger only grew when the department admitted spending $4 million on ads last year promoting its efforts to help veterans return to civilian life.
Reports detailed the troubles veterans encountered getting help and benefits from the department. Other studies suggested that wounded veterans would face poverty once they hit age 65.
Efforts to calm the situation failed. Tweaks to benefits and more money for mental health brought no respite. Fantino's chief of staff quit and was replaced by a staffer from the Prime Minister's Office.
Retired general Walt Natynczyk, the country's former top military commander, was also appointed as Fantino's deputy minister and senior civil servant.