The federal agency tells CBC 11 fishing operations in the area were deemed "suspicious" after regular monitoring of claims.
"It's important we do work to check the integrity of the EI system," says Alyson Queen, director of communications for Diane Finley, Minister of Human Resources and Skills Development.
Queen says 182 claims involving 102 individuals were reviewed. 44 people were cleared of wrongdoing after a year-long investigation by the Canada Revenue Agency.
The remaining individuals had claims denied. Ottawa is threatening to recover funds from them if allegations of misconduct are proven. The federal government says claimants were family members who did not maintain an "arms length" relationship with the employer.
The crackdown has sent shock waves through fishing communities in Northern Cape Breton where employment opportunities are limited to seasonal industries like fishing and tourism.
The EI investigation was revealed by the area's Liberal MP Mark Eyking. The Harper government accuses him of exaggerating the situation.
"There's been false information put out there. Mr. Eyking is trying to score political points on what are serious allegations," Queen tells CBC News.
Eyking questioned Finley on Wednesday in the House of Commons.
"It's my job to fight for the people I represent," said Eyking. "I'll continue to fight for those attacks on the good people of northern Cape Breton. Minister, what Atlantic fishing community is next on your hitlist?"