Thursday's announcement was met with disappointment and outrage.
"It's a travesty, as far as I'm concerned," said Tony Scott, a one-time resident of the former orphanage for black children. "We do feel, unfortunately, that racism is playing a role in this."
In March, police formed a special investigative team to look into the allegations and asked people to come forward.
Forty complainants were located in several provinces including Nova Scotia, New Brunswick, Quebec, Ontario and Alberta. The team visited the witnesses in each location.
But the evidence that was collected wasn't enough to support a charge, police said.
"The information obtained was unable to be corroborated to meet the threshold that would formulate reasonable and probable grounds to lay criminal charges, RCMP said in a statement.
Smith isn't convinced.
"We don't know each other and we're all telling the same story, so that's really strong evidence that this took place," he said.
Police noted that some of the allegations go back many years and some of the alleged abusers have died.
Though the investigation is now closed, police are still asking anyone else with information to come forward.
Demand for inquiry
Smith is convinced a public inquiry is needed to uncover the truth.
Tracey Dorrington-Skinner, another former resident, agrees. She calls the RCMP's decision "just another roadblock."
"This further goes to point out to the government why this public inquiry is so important to us," she said. "We want to get the government to understand that these atrocities did happen."
Justice Minister Ross Landry says it's too early to make a decision.
He said now that the RCMP decision is out, he needs "to look at this from that angle," but also do a "little further consultation."
The Home for Colored Children first opened in 1921 as an orphanage for black Nova Scotian children. More than 100 people have signed on for a class-action lawsuit, with many claims going back to the early days of the facility.
The home is currently a short-term centre open to all children.