The research suggests the rule, which outlawed bodychecks aimed at the head and checking from a player's blind side, has not led to lower concussion rates among pro hockey players since it came into force in the 2010-11 season.
The senior author of the work said the league should take another crack at the rule change, noting that as it stands the wording is too subjective and gives referees leeway not to enforce it.
"If player safety is the prime priority of the NHL in bringing this kind of rule in ... then they need to relook at this in a very serious way and adjust things," said Dr. Michael Cusimano, a neurosurgeon who heads the injury prevention research unit at Toronto's St. Michael's Hospital.
"If it isn't a priority, I could see them just leaving things the way they are and it's kind of a Band-Aid response to a major problem."
The NHL did not respond Wednesday to a request for comment on the article. And the NHL Players' Association declined to comment because it hadn't had a chance to review the study.
But another concussion expert applauded the work, saying Cusimano and his team had performed a service by exploring the impact of the rule change.
Dr. Charles Tator, a brain surgeon with Toronto Western Hospital, said the change's lack of impact has an effect not just in the arenas of the National Hockey League, but on rinks where kids who dream of making it to the NHL some day emulate their professional heroes.
"Professional hockey is still a bad influence on the amateurs," said Tator, who is project leader for the Canadian Sports Concussion Project at the Krembil Neuroscience Centre.
The study was published Wednesday in the journal PLoS One.
Cusimano and colleagues painstakingly put together data on reports of concussions and suspected concussions — based on reports of symptoms — from a variety of sources. Some of the information came from teams, others from media reports.
The information was gathered for both the NHL and the Ontario Hockey League, which has stricter rules on checks to the head than the NHL does. The OHL rule penalizes any hit to the head of another player, intentional or unintentional.
The data showed that there was no statistical significance in the incidence of concussions in the NHL in the 2010-11 and 2011-12 seasons compared to the 2009-10 season. That latter was the year before the NHL rule change went into effect.
The researchers estimated there were about 5.23 concussions per 100 games in the NHL regular season. Despite its stiffer rule, the OHL didn't have markedly different concussion rates, clocking 5.05 per 100 games in the regular season.
The analysis also showed that the type of hits outlawed by the NHL rule weren't actually the major cause of concussions.
About 28 per cent of interactions produced a concussion also generated a penalty call, said Cusimano. In that 28 per cent, the bulk of the penalties were for fighting. "And blindsiding, which was what the rule was initially was written about, was only 4.1 per cent of all those.... But four per cent of 28 per cent is a very small number."
"I wasn't totally surprised, but I was disappointed that we weren't able to show a difference," Cusimano said.
"Part of it's the way the rule's written. Part of it's the way the rule is enforced. Part of it's the penalties associated with the rule. And part of it is that concussions are also coming from other causes like fighting, that is still allowed."
The way the NHL rule is worded gives referees outs to avoid levying penalties for some of the hits, for instance in cases where players are deemed to have put themselves in a vulnerable position.
"So it's like his fault, because he put himself into a vulnerable position. And this highlights one of the major problems in sport and particularly in hockey these days. We victimize the victim even more, rather than looking at the game and the system and saying: 'What can we do to reduce these injuries?'" Cusimano said.
He suggested that if the league wants to get serious about protecting players, it has to raise the cost of concussion-inducing hits, both on the player who inflicts the injury, and on the team which sent him out to do it.
If the player who sidelined Pittsburgh Penguin captain Sidney Crosby for a year was forced to spend as much time off the ice for the injury, the culture of teams might start to change, Cusimano suggested.
"If there were more severe consequences to those who inflict that kind of injury — let's say that player was out for an equal amount of time as Crosby — that might have more impact," he said.
Tator estimated that hockey has moved only about 10 per cent of the way down the path it would need to take to make the game safe for amateurs and professionals.
"In terms of injury prevention, it isn't enough to enact regulations," he said. "The other half of the coin is enforcement. And if you really aren't strictly enforcing a rule, the rule is going to be ineffective."