"We've got two choices as a party," Hudak told reporters in his first comments on the challenges to his leadership that surfaced recently.
"We can choose to fight among ourselves and fall father behind and let the Liberals call an election and win government again, or we can join together and fight side by side."
Two members of the PC caucus, Frank Klees and Randy Hillier, have openly supported some grassroots Tories who filed a motion asking for a constitutional change so there could be a leadership review during a party policy conference in London next month.
Some Conservatives are upset the party took only one of five Liberal seats up for grabs in Aug. 1 byelections, after losing the long-time Tory seat of Kitchener in a 2012 byelection and blowing a huge lead heading into the 2011 general election.
But a defiant Hudak was still putting his own spin on the byelection results Monday, crediting "union muscle" with helping the NDP take the London West riding that many had expected would go Conservative and boasting about winning the first Tory seat in Toronto since 1999.
"We should neither be thrilled with the byelection results nor despondent," he said.
"We actually got the most votes of any party, the most votes in the city of Toronto."
Klees and Hillier, both of whom lost leadership bids to Hudak in 2009, say it'd be a mistake for him to ignore party members who want a leadership review.
"Let's have a confidence vote, and if that takes place I'm convinced that it will be in the best interest of the party, rather than allowing this thing to fester and build up pressure unnecessarily," Klees told The Canadian Press last week.
"If in fact what we're hearing is true, and that is that the majority of the membership are in fact supportive, then he will survive it."
Hudak noted that he got over 78 per cent support during the leadership review that was mandatory after the Tories lost the 2011 election, and said he's not willing to agree to face another vote in September.
"I went through that just last year and four out of five party members supported me continuing as leader," he said.
"Did I get everybody's support? No, it's the nature of the job."
Just back from vacation, the embattled Hudak insisted he wasn't worried about critics inside the party who are challenging his leadership.
"When you become leader of the Opposition, you go through a bit of a procedure where they give you really thick skin and they give you broad shoulders," he joked.
"If you think that this is tough, it's nothing compared to what we have to go through fighting the Liberals, the Working Families Coalition and the special interests that are running the province today."
Foreign Affairs Minister John Baird is one of several high profile Conservatives who have come out in defence of Hudak's leadership of the Ontario Tories.
Also on HuffPost