Pictures of Duffy's house in Cavendish captured earlier this month showed little sign anyone was living there. Tax records also showed he does not receive the credit given to provincial residents on his property tax.
Speaking to reporters at a ceremony marking Heritage Day in Charlottetown, Duffy said the Cavendish home was not occupied during the winter.
"In the dirty winter months I do what hundreds of other Islanders do, which is I rent a place in town," he said.
"You can't be out in the middle of a blizzard if you have a heart problem or a medical problem and get to the hospital."
Pressed for details on how many days a year he spends on P.E.I., Duffy said he would leave the questions of his residency to an upcoming report by Deloitte commissioned by the Senate.