Speaking to the Canadian media at the end of the meeting, the prime minister grinned when asked about his admonishment to Vladimir Putin to get out of Ukraine, though quickly checked himself.
"It is necessary for all of us to keep the pressure on Mr. Putin and his regime and to do so over the long term, to make it clear that it will not be business as usual," he said.
"We cannot have a major power in this day and age seize the sovereign territory of another country, and simply move on as if nothing's happened."
If the world community eases up on Russia for its annexation of Crimea earlier this year, he added, it will only whet Putin's appetite for similar aggression.
"We send the message, as I think we have in Canada, that whether it takes fives months or 50 years, we're not going to drop the subject until Ukrainian territory is returned to Ukrainians."
Harper described Putin's response to his smackdown as "interesting."
"Claiming they haven't done any such thing when we all know they have of course tells us that they really are on the defensive. If you're actually proud of what you've done, you'd go out and say so."
Harper's spokesman, Jason MacDonald, said the prime minister was speaking to a group of G20 leaders at a private leader's retreat on the opening day of the summit when Putin approached and extended his hand.
MacDonald said Harper told Putin: "I guess I'll shake your hand but I have only one thing to say to you: You need to get out of Ukraine."
A spokesman for the Russian delegation said Putin's response was: "That's impossible because we are not there."
Putin departed the meeting earlier than planned following other tense encounters with Western leaders as some of his fellow G20 members, including U.S. President Barack Obama, raised the prospect of yet further sanctions against Russia.
Putin left the summit, missing the official Day 2 lunch, following a meeting with British Prime Minister David Cameron and another strained encounter with French President François Hollande. The Russian president said he needed to get back to Russia to catch up on sleep ahead of a busy day on Monday.
In private talks that were reportedly "robust," Cameron warned Putin that Russia's entire post-Cold War relationship with the West was at risk due to the situation in Ukraine.
Hollande, meantime, has delayed a deal to supply two warships to Russia, angering Putin.
Australian Prime Minister Tony Abbott also lashed out at Putin for apparently flexing his military muscles by sending four Russian navy ships to stalk Australia's northern coast in the days leading up to the G20 summit.
Harper has been a vehement Putin critic for months, with Canada and Russia trading a number of retaliatory sanctions.
On the sidelines of the summit, one expert warned the Ukraine conflict could spark another world financial crisis due to the relentless back-and-forth economic sanctions between Russia and the West.
"You simply can't generate growth in Europe unless you get Russian troops out of Ukraine and get Russia to give back Crimea to Ukraine over the next five years," said Canadian John Kirton, co-director of the G20 Research Group.
"Unless Ukraine is solved the right way, you could get Russia canalizing or channelling a financial crisis."
On Day 2 of the summit, Harper met with top European Union leaders Jean-Claude Juncker and Herman Von Rompuy. He also sat down with Spanish President Mariano Rajoy Brey.
Harper also took part in G20 working sessions on global economic resilience.
The G20 leaders finalized a plan Sunday to boost global GDP by more than $2 trillion over five years by investing in infrastructure and increasing trade.
The communique issued at the conclusion of the summit says the leaders plan to jump-start growth, in part, by creating a global infrastructure hub.
The plan will purportedly help match potential investors with projects, and also help reduce the gap between male and female participation in the workforce by 25 per cent by 2025.
Abbott said countries will hold each other to account by monitoring implementation of their commitments to boost growth. He noted the leaders unanimously agreed that expanding global trade would directly benefit countries and people around the world.
Follow Lee-Anne Goodman on Twitter @leeanne25
Also on HuffPost