"We've focused too much on bulk collection just because there's a capacity to survey broad swaths of digital communication and collect it and store it, potentially indefinitely," says Adam Molnar, a Canadian security expert teaching at an Australian university.
But that collection may not only be harmful to privacy and civil liberties concerns, but ineffective as well, the Deakin University lecturer argues.
"Even in instances where we see an attack occur, these agencies are drowning in data and they're not even able to follow up on specific leads."
Molnar cites the 2013 Boston Marathon bombing and the recent Paris attacks as cases where information was gathered in suspects, "but it made very little difference."
In light of Canada's own attacks on soldiers, U.S. journalist Glenn Greenwald said Canadians should be asking tough questions.
"It raises a real question about why these Western intelligence agencies that are spending so much money on these very sophisticated means of surveillance can't find individuals who are planning attacks like that?" asks Greenwald.
The constitutional lawyer and author is famous for helping publish a trove of top-secret documents obtained by U.S. whistleblower Edward Snowden over the past two years. Revelations from the Snowden files have prompted debates about privacy and security around the world.
Greenwald was part of a team from the U.S.-based news site The Intercept who worked in collaboration with CBC News to analyze Canada-related Snowden files.
Those files included a 2012 presentation by a team at Canada's Communications Security Establishment (CSE), which is taxed with electronically monitoring security threats abroad. The presentation revealed CSE's sweeping access to videos, music and documents shared on free file-hosting websites through a project it codenamed Levitation.
Under Levitation, the electronic spy agency was sifting through up to 15 million uploads or downloads each day from around the world as part of a counterterrorism effort. But, according to the presentation, only 350 downloads each month triggered any kind of follow-up — an extremely small portion of the indiscriminately collected data.
The way the program worked was that the CSE tapped into collected metadata on those downloads. It then used the computer's IP addresses to cross-reference that through at least two wide-reaching databases of metadata held by Canada's spying partners to try to figure out a suspect's identity and to further monitor that person's online activity.
New legislation coming
Questions about the effectiveness of mass surveillance are being raised as the Canadian government plans to introduce new legislation Friday to give security agencies broader powers. The new rules come in the wake of two attacks on Canadian soldiers last year as well as a growing number of extremist incidents around the world.
Wesley Wark, a national security expert, says that no matter how many "interesting needles" come out of the haystack of online data, spy agencies still need to translate that to "usable intelligence" – meaning something they can act on.
"At the end of the day, one piece of good intelligence might be worth it all," says Wark, who is currently at the University of Ottawa.
In its 2012 presentation to its "Five Eyes" spying partners — the group that includes the U.S., U.K., New Zealand and Australia — the CSE mentioned two important successes from the Levitation project.
The first involved the discovery of an uploaded document that outlined the hostage strategy of AQIM, the North African branch of al-Qaeda. That strategy was "disseminated widely," including by the CIA to its overseas counterparts, the CSE presentation says.
Cyber analysts also unearthed a video of a German hostage from a previously unknown target. That hostage died in late May 2012, months after spies came across the video.
Edgar Fritz Raupach, an engineer working in Nigeria, was killed by his hostage-takers when local soldiers — who were unaware of Raupach's presence — attacked the captors' hideout in an unrelated operation.
Wark cautions that the document — as a presentation by CSE to its spying partners — is inevitably biased toward touting the most favourable results. Ultimately, he says, success in this business depends on whether the findings were timely, didn't consume too many resources and were useful.
"These Canadian documents suggest it can pay off," says Wark. "So, does it pay off? Is it proportionate to the resources we're putting into it? Are there different ways to do it?"
It is not known whether the Levitation project is still ongoing. CSE says it can't comment on details of the program, citing the Security of Information Act.
Julian Fantino, the associate minister of national defence, told CBC News in a statement that CSE's foreign signals intelligence have "played a vital role in uncovering foreign-based extremists' efforts to attract, radicalize and train individuals to carry out attacks in Canada and abroad."
"Our government will not sit idly by while terrorists use websites to attract, radicalize and train individuals who threaten our values and freedom."
As for the new bill coming Friday, Employment Minister Jason Kenney said the objective is to stop attacks before they happen by targeting what's being called incitement to terrorism.
Sources told CBC News that the legislation will give security agencies the ability to obtain and share information now subject to privacy limits, and make it easier for police to detain suspected extremists.
However, Liberal MP Joyce Murray says while there's been calls for the government to tighten up security, privacy concerns must not be forgotten.
"They need to also look at the provisions to protect individual privacy," said Murray. "And the government has failed to do that."
Murray says laws governing the CSE are 14 years out of date and don't touch on metadata.
The so-called data about data — which for email can include information such as recipients, subject lines and dates — falls outside the old laws because it isn't considered "private communication." Only the contents of an email or a conversation during a phone call are considered a communication.
Big topic in U.S.
While there has been relatively little debate in Canada weighing privacy concerns in the face of security fears, it's been a hot topic in the U.S. since most of the Snowden revelations involve CSE's counterpart, the National Security Agency.
Last year, a Washington-based non-profit analyzed 225 terrorism cases inside the U.S. since the Sept. 11, 2001 attacks and found that bulk collection of phone records by NSA had "no discernible impact" on preventing acts of terrorism.
The non-profit New America Foundation said the bulk collection of phone metadata — which includes phones called and call duration — had, in its view, only marginal impacts on preventing terrorist-related activities.
The organization said in most cases it was traditional law enforcement and investigative methods involving a tip or evidence that resulted in initiating action against an individual or group.
That finding came on the heels of a White House-appointed review committee that drew a similar conclusion. It said that much of the evidence that NSA turned up from tracking phone calls could have "readily been obtained" using standard court orders. It found that the phone metadata collection program was "not essential to preventing attacks."
For Molnar, the lessons from the U.S. are clear. "It tells us that [bulk collection] actually does very little in terms of identifying unknown suspects or actually detecting and preventing attacks before they occur."
Similar analyses on the effectiveness of gathering so much online metadata haven't been done since much less is known about the programs collecting them, says Tamir Israel, a lawyer with the University of Ottawa's Canadian Internet Policy and Public Interest Clinic.
Ultimately, the invasion of privacy is disproportionate to the benefit, he says.
Earlier this week, a report by Europe's top rights body said that mass surveillance programs are endangering fundamental human rights, including the rights to privacy, freedom of expression and freedom of religion.
The Parliamentary Assembly of the Council of Europe said it is"deeply worried" about the use of secrecy laws and secrecy courts — all of which is "very poorly scrutinized."
"In the long term, this type of unfettered surveillance is a really insidious thing that can really have very serious negative impacts on the way democratic institutions work," says Israel.
On mobile? Click here for a look at the step-by-step Levitation process
Also on HuffPost