In an interview airing Tuesday on CBC News Network's Power & Politics, Anders told host Evan Solomon "these people are Red Tories who are going to recruit Liberals and New Democrats because they know they don't have enough numbers in and of themselves."
Anders has been elected to represent the riding of Calgary West since 1997.
But David McKenzie, a Calgary lawyer who is part of the group challenging Anders, said the constituents are "disappointed" and "fed up" with their MP, even "embarrassed" by him.
During that same interview, McKenzie told Solomon "unfortunately, when Mr. Anders finds himself in the media it's generally not for good reason."
"Mr. Anders has had a hard time staying awake in the House of Commons, staying awake in committee meetings and I think his stance with respect to Nelson Mandela going back a number of years… doesn't jive with the views of Conservatives in Calgary West," McKenzie said.
Anders blamed a past car accident for falling asleep inside the House of Commons and apologized to veterans after they criticized him for falling asleep during a veterans affairs committee meeting.
The Conservative backbencher was the only MP opposed to giving Mandela honorary Canadian citizenship in 2001.
Red vs. Blue Tories
Meanwhile, Anders said the challenge to his nomination has nothing to do with his views or his credibility as an MP.
He said it's simply "an age-old battle" between Red and Blue Tories.
"The Red Tories are like a flu that just won't go away," Anders said.
McKenzie fired back saying "It's a battle for his job."
"It's not about Red and Blues from my perspective, it's about representation. Everybody that I'm talking to is as Blue as Mr. Anders thinks himself to be, including myself."
"It's about changing our representation so that we've got somebody who better reflects the viewpoints of the residents in Calgary West," McKenzie said.
Former Alberta cabinet minister Ron Liepert is also part of the group challenging Anders' nomination, although he has yet to say whether he will run against Anders.
"Liepert is a well-known Red Tory… and Red Tories don't stand a chance unless they recruit Liberals," Anders said.
McKenzie said the group is recruiting voters who voted Conservative but have become dissatisfied with the representation in the Commons.
This is not the first time voters have openly challenged Anders.
In 2012, a Calgary man upset with Anders over comments he made after introducing a private member's bill launched a petition to see him removed from office.
Geoff Norquay, a Red Tory who worked in the Prime Minister's Office under Brian Mulroney, said the battle between Red and Blue Tories was put to bed when Stephen Harper was elected leader of the Conservative Party.
Appearing on a panel, Norquay did not take sides but told Solomon the nomination challenge for the new riding of Calgary Signal Hill in 2015 is shaping up to be "a great battle."
No date has been set for the nomination in Calgary Signal Hill.
Also on HuffPost