The group has launched a website — with the motto "It's time to do better" — with the goal of shutting out Anders when the party next nominates a candidate in his Calgary riding.
The website notes that 2014 marks the 17th year Anders has been an MP and says it's time for a change.
"The team is in place to make that happen," reads the website.
"More details will be announced soon. We would like you to join the movement for change by purchasing a membership and providing us with an email address so we can keep you informed."
Anders, 41, has represented Calgary West since 1997. The riding name is changing to Calgary Signal Hill under federal redistribution.
"It's been a real grassroots thing. I haven't run into anyone who has been satisfied with the representation Calgary West has had at the federal level. It surprises me how impassioned people have become," David McKenzie, a Calgary lawyer who is chairing the campaign to replace Anders, said Monday.
"Mr. Anders, when he's in the media, it's generally not for reasons that people are proud of and it's time for change."
Anders did not respond to a request for comment.
He has faced a number of challenges during past nominations, including attempts by now Premier Alison Redford in 2004 and former Alberta member of the legislature Jocelyn Burgener in 2000.
He describes himself as a social conservative who is pro-life.
Anders opposed honorary Canadian citizenship for Nelson Mandela in 2001 and labelled him a communist and a terrorist.
He is vitriolic in his dislike of China. He once compared the 2008 Beijing Olympics to the 1936 Berlin Games.
In 2012, Anders was dropped from the Commons veterans affairs committee after he lashed out against a veterans support group which had criticized him for falling asleep during a committee meeting.
He later apologized for saying his critics were NDP "hacks.''
"My personal favourite is falling asleep. It does cause one to start wondering about what we are paying for in terms of our representation. Just being there is not enough," said McKenzie.
"The thing that hit my own radar was Rob Anders being the sole MP who refused to assent to granting honorary citizenship to Nelson Mandela, which required a unanimous vote. I just thought it was embarrassing and that's the type of feeling I'm getting from my neighbours."
McKenzie acknowledges that Anders is very well organized.
The group's candidate is to be announced by the end of this month.
Former Alberta cabinet minister Ron Liepert admits he is part of the group hoping to replace Anders.
"I think at the end of the day we will have a nomination and I will have to make a decision," Liepert said Monday in a phone interview from Palm Springs, Calif.
"I can't deny that I think it would be helpful to have someone who has a bit of name recognition and some experience in public life to challenge this fellow."
Liepert was involved with Redford's attempt to get the nomination in 2004 which he says is the last time there was an open nomination in the riding. He said he and many other Conservatives have let their federal memberships lapse as part of a "silent protest" against Anders.
He appears to be leaning toward putting his name forward.
"It wasn't part of my game plan because flying back and forth to Ottawa isn't something I had in mind, but at the end of the day, if we don't have a solid strong candidate in place...he will be the candidate again."
Also on HuffPost