Jacques Fauteux, Fantino's chief of staff and top adviser, revealed Thursday that he is leaving, effective immediately.
It is unclear at the moment who will take over for him.
His departure comes just days after retired general Walt Natynczyk, the country's former top military commander, took over as deputy minister, the department's top civil servant.
Natynczyk's appointment, announced by the prime minister at the end of October, was seen as a move by the Conservatives to get a handle on a file that has caused the government a lot of grief.
Angry veterans promised again this week to campaign against the Conservatives in next year's election, citing grievances about benefits and the closure of Veterans Affairs offices.
The minister was involved in a testy exchange with ex-soldiers on Parliament Hill last winter and also walked away from an angry military wife last spring — two televised incidents that made him appear less than sympathetic to the plight of the wounded and their families.
Fantino, the former commissioner of the Ontario Provincial Police, switched from policing to politics in 2010, agreeing to run for the Conservatives in the politically important riding of Vaughan.
He received a junior spot in cabinet almost immediately after, unusual for a rookie politician but reflective of the high value the Tories placed on having a cabinet minister from the vote-rich Toronto suburbs.
He became veterans' affairs minister after a cabinet shuffle in 2013 but frustration has been mounting about his seeming inability to quell problems in the department.
Among the upcoming hot spots is the release of an auditor general's report into the state of mental health programs for veterans.
Fauteux, a veteran member of the military, told colleagues and friends Thursday that his departure is bittersweet given the events three weeks ago and the murder of a Canadian soldier at the foot of the national war memorial.
He said he'll miss the opportunity to make a difference, but thanks Fantino, Prime Minister Stephen Harper and Natynczyk, among others, for the opportunity to serve.
Fantino, in turn, expressed his gratitude to Fauteux for "his dedicated and loyal service to me during my time as Associate Minister of Defence and Minister of Veterans Affairs."