If there is a crux to the senator's testimony, which reached its sixth day on Tuesday, his animated description of his dealings with Stephen Harper's team in February 2013 is it.
"I knew they were building a scaffold and everywhere I looked I saw people who were betraying me, leaking stuff to the media that wasn't true," Duffy told the court.
"I was all alone..."
Sen. Mike Duffy, former prime minister Stephen Harper. (Photo: PMO/court evidence)
Duffy has pleaded not guilty to 31 charges of fraud, breach of trust and bribery. Some of the charges have to do with his filing living expenses for time spent in his longtime home in the Ottawa area, on the premise his primary residence was in Prince Edward Island.
The bribery charge relates to the allegation that he was a participant in a scheme to have Harper's one-time chief of staff Nigel Wright secretly repay $90,000 in expenses, while Duffy told the public he had done so.
Duffy's defence is that he was coerced into agreeing to the scheme by powerful Conservatives, including Wright and Harper. He said he made his case to the former prime minister after a February 2013 caucus meeting, arguing that telling the public he had made a mistake with his expenses would ruin his reputation.
"I know it seems unfair, I know you didn't break the rules, but the rules are inexplicable to our base (of political support) and therefore you're going to have to pay the money back," Duffy says Harper told him.
"Nigel will make the arrangements."
A few days later, he and Wright would have a testy phone call about how Duffy should admit making a mistake and repay the expenses. The senator was continuing to dig in his heels _ he felt he could prove he spent enough time on P.E.I., with other weeks spent on the road trying to raise support for the Conservative party.
"I had fought and I had fought and I had fought, and I tried every kind of resistance, but when they pulled that knife out and held it over my head, I felt I had no other choice."
Duffy alleges that Wright warned him that a powerful, Conservative-dominated Senate committee was poised to issue a report saying he wasn't constitutionally eligible to sit in the Senate for P.E.I., because of his living arrangements.
"I realized that after everything that I had seen, everything that had been done to me, this seemed like a very live option, that they wouldn't hesitate to do this to me," said Duffy.
"In my mind that night, I thought, this is it, I'm cooked, I'm cooked."
When Duffy asked Wright to produce the analysis that said he had broken the rules, he said Wright became angry.
"He said, 'if you keep defying the prime minister, you'll end up like Patrick Brazeau, out of the caucus and probably out of the Senate.' And I said, 'For what?"' Duffy recounted, referring to Brazeau's ouster a few weeks earlier following an assault charge.
"He said, 'Listen to me, you're defying the prime minister ... you're going to do this.' It was quite snarky."
Wright testified in August that he had indeed pressured Duffy to repay his expenses, but that he had not raised the threat of expulsion from the Senate. On the contrary, Wright said he continually reassured Duffy that he was on a solid constitutional footing.
After the call with Wright, Duffy said he received more calls from Conservatives pressuring him to give in. Those conversations, combined with a new review of residency status partly triggered by then Conservative Senate leader Marjory LeBreton, caused Duffy to finally acquiesce.
"That was it. That was it. I had fought and I had fought and I had fought, and I tried every kind of resistance, but when they pulled that knife out and held it over my head, I felt I had no other choice," Duffy said.
From there, negotiations began between Duffy's lawyer, Wright and Harper's lawyer. The Crown has alleged that Duffy made a number of demands around the repayment, including that he not be out of pocket for the expenses and that he be withdrawn from an independent audit by the firm Deloitte.
Duffy told the court that he never asked for money, that he told Wright he didn't have any funds to see if that would finally make him abandon the scheme.
Instead, he said Wright told him that because he had spent so much time travelling and working for the party, that the party would defray the repayment.
Wright ultimately wrote a cheque for $90,000 in late March 2013. It would emerge in the media two months later. Wright would leave the PMO, and Duffy would eventually be suspended from the Senate.
ALSO ON HUFFPOST: