Joan Jessome of the Nova Scotia Government and General Employees Union said Wednesday that 22 of the 23 staff in the Justice Department have indicated they will not move to New Waterford from Halifax when their positions are transferred there.
"They're really torn between following the work that they've been trained to do and they love versus keeping their families stable," she said.
"Many of them are choosing to stay where they're at."
The province announced in April that the workers with the department's maintenance enforcement program would be moved to the Cape Breton community.
Another 68 jobs with the Fisheries and Agriculture departments are slated to be moved to Digby, Shelburne County and Truro.
Jessome says only three of the 21 people affected in the Agriculture Department have indicated so far that they will move to Truro, while one has opted to receive severance.
Nine of the 27 unionized Fisheries workers have said they will not relocate.
The workers have until July 4 to decide if they will move, seek another government position or take severance.
The union represents 71 of the affected workers. Of those, 32 hadn't yet told their employer what they would do.
Jessome said the loss of staff will cost departments expertise and experience.
"We're disturbed by it because of the expertise (in Fisheries). These are very specialized jobs and they've built up a relationship with a billion-dollar industry in this province," she said.
"There's a workload issue for the ones who are left behind."
Deputy premier Frank Corbett has said the government will hire or transfer people to replace those who don't relocate.
The province has not put a price tag on the overall cost of the moves, but the union has said employees who won't transfer are eligible under their collective agreement for two opportunities of "reasonable employment" elsewhere in the government.