Vernon-Monashee MLA Eric Foster was cited in a confidential letter from the AG's office last year over inadequate documentation for a 2009 renovation of his constituency office which cost $67,000.
Foster says he has done nothing wrong and declined to comment further on the issue on Thursday.
But his critics say the whole issue raises doubts that Foster could be objective when it comes to Doyle's appointment.
Conservative Leader John Cummins says it is clear Doyle was causing problems for the Liberal government for some time and that might explain why they want him gone.
"I think Mr. Doyle has earned the right, if he so desires to stay here. He's earned the right to have his appointment continued. And I think B.C. residents should be putting pressure on the Liberal government to make sure that's the case."
Bigger issue at play
Former premier Bill Vander Zalm also says he thinks there is a bigger issue at play.
"The safest thing for an auditor general, given this system as we know it today, is to not do too good a job," Vander Zalm told CBC News.
Vander Zalm says leaving politicians responsible for appointing and reappointing people who are supposed to be watching over them doesn't work.
Federal auditors general are appointed to 10 year terms, but in B.C., they are appointed to six year terms, and Vander Zalm says that becomes a problem when the job comes up for renewal.
"I'm not pointing fingers at the Liberals or the NDP or Socreds before that. It's just the system is no good and this issue has brought it to the fore," he said.
He says one solution would be for politicians to recommend candidates for auditor general and let the B.C. Supreme Court make the final decision.