During the nearly hour-long, closed-door meeting at an east Toronto police station, blocks away from the scene of the shooting that left two dead and 23 others injured, the two exchanged a number of "specific suggestions" for curbing violent crimes, Harper said.
He would not elaborate on what the measures may be, but said they will be crafted in the next few months.
The mayor of Canada's largest city agree with him on the importance for tough sentences for those caught with illegal firearms, Harper said.
"I think these events in Toronto underscore why these penalties are essential," the prime minister said later in the day at a funding announcement at GM Canada in Oshawa, Ont.
"This is not a theoretical problem."
Harper lamented that the courts were "attempting to strike down" some stiff punishments, appearing to refer to recent decisions declaring mandatory minimum sentences for gun crimes unconstitutional.
"We’ve also been working together at trying to make sure we can make some of these gun penalties stick before the courts," said Harper.
"We discussed a range of enforcement measures, a range also of the criminal justice measures that are before Parliament and the necessity of making those stick."
Earlier this year, the federal government introduced more tough-on-crime measures in an omnibus crime bill that saw mandatory jail sentences for offences such as drug trafficking, violent offences and sex crimes.
It also eliminated conditional sentences for crimes such as sexual assault, manslaughter and drug trafficking. The bill also dictated stiffer penalties for young offenders.
The prime minister pointed to several initiatives by the government that showed it continues to make fighting crime a top priority.
He said it has invested hundreds of millions of dollars in border security to try to stop the smuggling of illegal guns from the U.S. and has tried to introduce new legislation that will make it easier to deport non-citizens who commit crimes in Canada.
A Conservative MP also recently put forward a private bill aimed at harsher sentences for those caught trying to recruit new gang members, said Harper.
The prime minister also added that the government has always supported programs to help at-risk youth, but did not provide any specifics.
Both Harper and Ford immediately drove off following the meeting, and didn't answer reporters' questions, though the prime minister took questions in Oshawa later.
In a statement released hours later, the mayor called the meeting "productive" and said he was looking forward to working on anti-crime initiatives with Harper.
"This is the beginning of ongoing work to make sure we have the tools in place to better prevent gang violence and protect the public from criminals," Ford said in the statement.
Outside the meeting, a number of community members were disappointed that they didn't have a chance to speak with the two leaders.
Rose Marie Riley said she would've pleaded for a ban on handguns. Her brother, Kevin Williams, 33, was gunned down in March 2010 outside his apartment in the same neighbourhood as the barbecue shooting.
She said the community needs to work together to end the violence.
"It starts with us," said Riley. "Be more open, if there is something that is going on, you need to report it to the police."
Ontario Premier Dalton McGuinty also spoke Tuesday with Harper about problems with guns and gangs in Toronto.
He said he suggested the creation of a task force to deal with federal prosecutions related to illegal handguns and what more could be done to stop the flow of the weapons into Canada from the U.S.
"The fact of the matter is most of the guns that end up in the hands of young criminals are illegal guns and they're coming from south of the border," said McGuinty. "I think we're all seized with a greater sense of urgency."
Ford met with McGuinty and Toronto Police Chief Bill Blair on Monday and secured permanent funding for anti-violence programs from the province.
McGuinty pledged $12.5 million in funding that was set to expire next year for specialized anti-violence police units, but did not give Ford the $5 million to $10 million the mayor wanted to hire extra police officers.
These talks came the same week that Statistics Canada reported that the country's crime rate is at its lowest in the last 40 years.
The agency said although the overall homicide rate was up seven per cent to 598 homicides in 2011, Ontario saw record lows in murders last year.
Also on HuffPost