Williamson told delegates at a conference in Ottawa that it makes no sense to pay "whities" to stay home while companies bring in "brown people" as temporary foreign workers.
By Saturday, the New Brunswick Southwest MP had apologized on Twitter but the criticism has not halted.
Williamson is also being criticized by his fellow Tory MPs.
Tobique-Mactaquac Conservative MP Mike Allen told reporters that it was important that Williamson apologized for the remarks.
“It’s always good when someone apologizes for saying something dumb," he said.
Tory MP Deepak Obhrai, the parliamentary secretary for foreign affairs and international human rights, also had tough words for his colleague on Twitter.
He said he was “very disturbed” by the comments.
“Foolish statement damages all of us. Years of hard work down the drain,” he tweeted.
Williamson is a former director of communications for Prime Minister Stephen Harper.
The MP said in a statement that the comments are the biggest mistake of his political career.
The comments were resonating in his riding on Monday.
Esthela Pyett, the president of the New Brunswick Filipino Association, said she was “hurt” by Williamson’s comments.
Pyett said the MP has helped many Filipino residents in the riding, but he clearly chose the wrong words at the conference.
“I think that's what he meant by the browns are coming here to work and they really work hard. And at the same time they pay taxes,” she said.
Williamson was also criticized by the riding’s Liberal candidate for the upcoming federal election.
Karen Ludwig said the MP’s language is totally inappropriate.
“When Mr. Williamson believed he was only speaking to Conservatives, he shamefully sought to divide Canadians,” she said in a statement.
“This is completely unacceptable. Our riding needs attention from Ottawa, but not like this.”