Answering questions from reporters during a joint press conference with Peruvian President Ollanta Humala Tasso, Harper said he had been through a "range of emotions" since learning about how Duffy's repayment happened.
"I'm sorry, I'm frustrated, I'm extremely angry about it," he said.
"I learned about this after stories appeared in the media last week speculating on the source of Mr. Duffy's repayment," he said. "I think what's more important is not simply that I did not know, but that I was not consulted. I was not asked to sign off on any such thing, and had I obviously been consulted or known, I would not have agreed with it."
"My belief here was reasonable, and what I think anybody would have expected, that when it was said that Mr. Duffy had repaid his expenses, that indeed he and not someone else had repaid his expenses," Harper said.
"In terms of my own office, it was Mr. Wright’s money, it was his personal money, that he was repaying to the taxpayer on behalf of Mr. Duffy," he said. "It was his personal decision and he did this in his capacity of chief of staff, so he is solely responsible and that is why he has resigned.”
"I know Mr. Wright assisted him or did this for him because he wanted to see the taxpayers reimbursed. That's the right motive, but nevertheless it was obviously not correct for that decision to be made without my knowledge or without public transparency."
"That is why I accepted the resignation of my chief of staff," the prime minister said. "My point is on this that there is accountability when these things happen."
"We will certainly look at our systems to see what we have to do to better manage, or better yet prevent, any of these kinds of things in the future."
- TIMELINE:Key moments in the Senate expense scandal
Harper was meeting with business leaders and Peruvian politicians Wednesday as part of a four-day trip to South America focused on trade and bilateral relations.
Wednesday's press conference was the first opportunity for reporters to question him directly about the expenses controversy.
Harper's chief of staff resigned on Sunday after giving Duffy a personal cheque for $90,000 so the former journalist could pay off ineligible expenses.
The prime minister told a Conservative caucus meeting Tuesday, before travelling to Peru, that he was "very upset" by the conduct of some senators and his own office, but did not take any questions from journalists, despite some attempts at shouting questions from the back of the room.
Air transport agreement reached
Harper discussed a number of issues Wednesday morning with the Peruvian president, including shared economic interests such as mining and the framework for a new air transport agreement with Peru, according to a joint statement.
"Both leaders affirmed the importance of bilateral trade and investment and their commitment to trade liberalization and open markets," the statement said.
The two leaders also discussed the Pacific Alliance — a free-trade area made up of Mexico, Chile, Peru and Colombia — ahead of the group's leaders' summit in Colombia on Thursday.
Canada, which has bilateral free-trade agreements with all four, gained observer status on the Pacific Alliance in 2012 and is considering joining.
Harper is putting the new "extractive industries" focus of his aid and foreign policy into practice as he talks about Canadian mining companies and their role in developing countries such as Peru.
The new approach was rolled out last fall, and aims to align Canada's aid spending more closely with its commercial interests — to much consternation from aid groups that fear Canadian business promotion will take precedence over poverty reduction.
At the same time, the policy is meant to ensure Canadian investors uphold high standards when it comes to labour and environment in developing countries.
Peru is one of Canada's key recipients of foreign aid, and Canadian mining companies have a large presence in parts of the country known for social unrest. So Peru — as well as Tanzania — is a key testing ground for the extractive industries orientation of foreign policy.
"It's still very conceptual," said Ottawa-based trade analyst Laura Dawson.
Canadian direct investment in Peru was $6.9 billion in 2012, much of it in the natural resources sector.