And while he blamed volunteer Reg Bowers for problems with his 2011 election campaign, Bowers said he still believes in Penashue.
Bowers, however, resigned his seat on a federal board shortly after speaking to CBC News.
Penashue said in a statement that he's proud of his record as an MP and a minister in the Conservative government since he was elected to represent Labrador in 2011.
"Due to mistakes that were made by an inexperienced volunteer in filing the Elections Canada return from the last campaign, I appointed a new official agent to work with Elections Canada to make any needed amendments to my campaign return," he said in the statement.
"During the examination, we became aware that there were ineligible donations accepted by the former official agent."
Penashue said he didn't know about the problems with the return, which included incomplete records.
'Right out of the blue'
Bowers was appointed by the Conservative government after the election to sit on the board of the Newfoundland and Labrador Offshore Petroleum Board.
CBC News has reported Penashue may have overspent his election limit and records suggest he may have accepted a corporate donation, which is illegal under Canadian law.
A spokesman for the Conservative Party says Penashue has "reimbursed the Receiver General [the federal government] for the ineligible contributions received" and confirmed that they added up to $30,000.
Before the Newfoundland and Labrador Offshore Petroleum Board announced he had resigned, Bowers told CBC News that he was surprised by Penashue's resignation.
"This is right out of the blue," he said.
"I found out when you did."
Bowers said he had been considering whether to resign from the CNLOPB and that he'd been trying for weeks to find out what's going on with the review by Elections Canada. The Conservatives replaced him in 2012 with their own chief financial officer, Sandra Troster.
It's a sad day, Bowers said.
"I don't want to be a distraction for Peter... I worked so hard to get that man elected."
"I still believe in Peter Penashue."
'Vindication' for voters
The 2011 Liberal candidate, who lost to Penashue by 79 votes, said Penashue stepping down is vindication for anyone in Labrador who called for another election.
"Nobody resigns a cabinet post if they feel they've done nothing wrong," Todd Russell told CBC News.
"This is an opportunity for the people of Labrador to have their say with fairness."
Russell said he needs to take a few days to consider whether he'll run for the seat he held from 2005 to 2011. He is now president of Nunatukavut, a group representing the Métis in Labrador.
A spokesman for the Conservative Party confirmed Penashue will be their candidate in the by-election, for which Prime Minister Stephen Harper has to set a date.
"If Newfoundland and Labrador want to continue to have a strong voice within government they need to re-elect him as the member of Parliament for Labrador," Fred DeLorey said in an emailed statement.
Airline wrote off thousands of dollars
In October, CBC News reported that Elections Canada records showed Penashue's campaign couldn't afford the flights he took around the large, sparsely populated riding, so Provincial Airlines wrote off $17,000 in plane tickets. Corporate donations are illegal in Canada.
The election campaign records, which each candidate is required by law to file, show repeated cash flow problems with Penashue's campaign. Aside from the flights that were written off, his campaign bounced cheques and finally took a $25,000 loan from an Innu group run by his brother-in-law.
CBC News calculated Penashue had overspent his campaign limit by about 21 per cent.
Election records show the campaign and Innu Development Limited Partnership hadn't set an interest rate for the loan until months after the election was over. Loans to election candidates must be subject to interest or the interest is considered to be a contribution.
CBC News later reported that a deposit slip included in the records suggested Pennecon, a St. John's, N.L.-based construction company, donated to his campaign. The deposit slip included a single entry for Pennecon but receipts were issued to several members of the company's board.
A statement from Prime Minister Stephen Harper thanked Penashue for his service and announced Transport Minister Denis Lebel would take over Penashue's previous role as intergovernmental affairs minister.