Duffy was so concerned, he asked Harper if he could represent Ontario, the province where he had lived for three decades.
But sources say the prime minister needed to fill a P.E.I. seat and insisted Duffy represent the Island instead.
The Prime Minister's Office would offer no comment beyond saying the matter is currently in front of the courts.
CBC News has put together some of the evidence tabled in court, and what emerges is the picture of a senator who was scrambling to deal with issues of residency in the days after being named.
The Constitution says a senator is eligible to sit in the Senate if he owns $4,000 worth of property in the province he represents and is a "resident" of the province.
On Dec. 22, 2008, Duffy was named a senator along with 17 others.
The next day, Duffy met with members of the Senate administration. Officials in court have testified they meet with new senators to walk them through some of the rules and guidelines around residency, and they did the same with Duffy.
While Duffy made his first per diem claim on that day, the documents filed in court show he did not fill out a residency declaration until days later, on Jan. 6, 2009.
On Dec. 29, Duffy travelled back to P.E.I., which he subsequently claimed as his first official trip as a senator.
Duffy's diary, which has been entered as evidence, shows that on Jan. 2 he and his wife, Heather, got P.E.I. driver's licences for the first time since the '70s.
On that same day, Duffy wrote in his diary that he spoke to "Dave Penner of PMO and Paul Belisle, Senate clerk on property documentation."
Penner was the head of appointments with the Prime Minister's Office at that time. Neither he nor Belisle returned calls to CBC, but Belisle is on the Crown's list as a potential witness.
Duffy returned to Ottawa and met with Belisle on Jan. 5. In his diary Duffy wrote that the meeting was in regard to "property documentation."
The next day he signed his primary residence declaration, claiming his primary residence was the cottage in Cavendish, P.E.I.
Senate officials have suggested in court testimony that signing this declaration was what allowed Duffy to tap into the $20,000 worth of living expenses he subsequently claimed.
On the same day Duffy signed the declaration, his diary shows he visited the office of then Senate leader Marjory LeBreton.
From that meeting, a memo was crafted for senators Duffy and Pamela Wallin from Christopher McCreery, a political aide to LeBreton.
In the memo, McCreery reassured the senators that "so long as a senator owns property in his or her province of appointment, then they are allowed to sit as a senator from that province, even if they live in Ottawa 99 per cent of the time."
And further, "The Senate has never disqualified anyone for being a 'resident' of their province of appointment, providing they own property there.'"
Eligibility raised in question period
The issue of Duffy's eligibility was raised during question period on Tuesday.
Opposition NDP Leader Tom Mulcair asked about when Duffy had signed his declaration of residency. "The prime minister says Duffy signed it before being named. It is not true. Can the prime minister please tell Canadians which declaration he was referring us to, and when exactly did Mike Duffy sign it?"
The prime minister did not directly respond to the question, but later said he would leave the matter to the courts.
As it turns out, that January memo may have alleviated Duffy's concerns about residency for a time, but not entirely.
In an email shown in court, later that month Duffy requested his assistant to ask Senate finance to make sure his tax deductions are for P.E.I. and not Ontario, even though this is where senators are taxed.
"Even if that is the case, I feel I must pay P.E.I. taxes, to reinforce my status as an Islander," Duffy wrote.
Also on HuffPost