The 6-3 ruling, penned by Chief Justice Beverley McLachlin, said the statute was unconstitutional as it upheld a 2013 Ontario Court of Appeal ruling that labelled the law cruel and unusual.
The court said the mandatory minimum sentence could ensnare people with "little or no moral fault" and who pose "little or no danger to the public." It cited as, an example, a person who inherits a firearm and does not immediately get a license for the weapon.
"As the Court of Appeal concluded, there exists a 'cavernous disconnect' between the severity of the licensing-type offence and the mandatory minimum three-year term of imprisonment," McLachlin wrote for the majority.
Justice Minister Peter MacKay said in a statement that the government will review the decision to determine "next steps towards protecting Canadians from gun crime and ensuring that our laws remain responsive."
"Our government will continue to be tough on those who commit serious crimes and endanger our communities."
McLachlin also took aim at the government's core justification for tough sentencing laws, which it says is to keep Canadians safer.
"The government has not established that mandatory minimum terms of imprisonment act as a deterrent against gun-related crimes," she wrote. "Empirical evidence suggests that mandatory minimum sentences do not, in fact, deter crimes."
Eric Gottardi, head of the Canadian Bar Association's criminal justice section, applauded the ruling because he said there is no solid evidence that stiffer sentences deter crime.
"In a climate where almost every law that is coming out of the government in the past 10 years has had some kind of mandatory minimum sentence attached to it, it is a pretty strong message to the government that their internal checks for what is constitutional clearly aren't working."
The case was focused on gun crimes, and will not affect other mandatory minimum sentencing laws on the books for possession of marijuana plants, sexual assault, child pornography and fraud and theft, Gottardi said.
Gannon Beaulne, a Toronto lawyer who has written in favour of mandatory minimum sentences, said the Supreme Court overstepped its jurisdiction, which is becoming a pattern.
"The rule of law is aided by any legislative policy that makes it easier to determine what sentence you're facing," he said. "It impinges on Parliament's jurisdiction to have a court of this country step in and erode that jurisdiction."
Justice Michael Moldaver echoed that sentiment in the dissenting minority opinion.
"In my view, sending our elected representatives back to the drawing board... would impede the goals of deterring and denouncing the unlawful possession of deadly weapons and keeping them out of the hands of those who would use them as instruments of intimidation, death, and destruction," Moldaver wrote.
The ruling struck down both the three-year mandatory minimum for a first offence of possessing a loaded prohibited gun, as well as the five-year minimum for a second offence.
The new gun sentencing rules were enacted in 2008 as part of a sweeping omnibus bill introduced by the federal Conservatives.
The Ontario and federal governments argued the move was an effort to combat a serious danger posed by the proliferation of handgun possession cases.
In one of the appeals at issue, a young Toronto man with no criminal record was sentenced to three years after pleading guilty to possession of a loaded firearm.
The judge said that without the mandatory minimum, he would have sentenced Hussein Nur to two-and-a-half years.
In the second case, Sidney Charles pleaded guilty to firearms offences after he was found in his rooming house bedroom with a loaded and unlicensed semi-automatic handgun. He was sentenced to five years because he had two previous convictions.
While the justices said the law is too broad in a general sense, they said the sentences were fitting in the two cases at issue.
The Supreme Court has clashed with the Conservative government on several key policies, although it recently sided with Ottawa over the destruction of gun registry data, which Quebec sought to preserve.
That win for the Conservatives came after several losses at the Supreme Court.
Also on HuffPost