Duffy spoke out Thursday, his first public comments since resigning last week after it was revealed he had made inappropriate expense claims and then paid them off with a $90,000 "gift" from Stephen Harper's chief of staff.
A number of Conservative MPs, including Heritage Minister James Moore, have said Duffy should quit his $132,000-a-year appointment.
But the former broadcaster, pursued by a gaggle of reporters and TV cameras out the Senate's front doors, literally blew out air in apparent exasperation when asked if he felt he should resign.
"I'm doing my job. So I'll see you at work next week," Duffy said.
So he'll remain on the job?
"Of course, I'm a senator, why wouldn't I?" Duffy responded.
A Harper appointee to the upper chamber straight out of his CTV job covering politics, Duffy has spent the past four years as the star attraction at Conservative party fundraisers across the country.
But media investigations last year began unravelling large expense claims for a "primary residence" in P.E.I. that turned out to be a seasonal cottage. Duffy has lived in Ottawa for decades.
CTV recently reported that Duffy had arranged a deal through the Prime Minister's Office to have a Senate audit "go easy" on him. That led to the revelation of a $90,000 personal cheque to Duffy from Nigel Wright, Harper's top adviser.
Since Duffy's resignation, fellow Conservative Senate appointee Pamela Wallin has also "recused" herself from the party caucus over questions about her expenses, and Wright has resigned.
The RCMP has asked for documents in the affair, according to the Senate Speaker; the parliamentary ethics commissioner is taking a look; and the Conservative-dominated Senate has also sent Duffy's expenses back to the same internal economy committee that originally cleared him of wrongdoing.
Duffy said he has not been contacted by the RCMP but will co-operate with police.
"I think Canadians have a right to know all of the facts, and I'm quite prepared, at the appropriate place and time, to give them the whole story," he said.
Asked if the prime minister knew about his secret deal with his chief of staff, Nigel Wright, Duffy said he had "no idea" but then began to offer more: "I would find, uh, uh, I just don't know."
And on other questions, such as the various versions of whether he got a bank loan or a gift from Wright, or both, Duffy did not clear the air.
"That's why we need a full and open inquiry, so that it all gets aired."
When it was suggested that Duffy himself could provide those answers immediately, he responded, "yeah, and I'm not giving them to you today."
Also on HuffPost