Dr. Aubrey Levin was convicted on three counts of sexual assault by a jury in January 2013 and sentenced to five years in prison. But that jury could not reach a verdict on four other charges.
The Crown retried Levin on two of them and a new jury, which began deliberations on Wednesday, was again unable to reach a consensus.
Alberta Justice Beth Hughes told court she had no choice but to declare a mistrial.
The Crown said it is considering whether to seek a conviction a third time.
"We're going to strongly consider it. It's one of those things that's been a serious enough case that we've marshalled a lot of resources for two trials," said Crown prosecutor Shane Parker.
"We're going to take a lot of input as well from our two complainants, who by now have testified three times, once at a prelim (pre-trial hearing), and two trials, so their input is something we're certainly going to want to have and we're going to have to take a look at the big picture as well."
Levin, who is 75, has asked Alberta's Court of Appeal for a new trial on his earlier convictions and has remained free on bail.
Parker said a new trial date could be set on May 9, but a decision to go ahead could depend on what the Appeal Court decides.
"The Court of Appeal decision, I'm sure, will be a major factor on what's going to happen here, as to whether there will be a third trial or not," he added.
"It has happened before but it is rare. It's happened unfortunately in this case for Dr. Levin, because now this is going to continue to hang over his head and it's obviously unfortunate for the two complainants who may have to testify yet again."
The original allegations against Levin came to light in 2010 after one of his patients came forward with secret videos he had recorded during court-ordered sessions with the psychiatrist.
The patient was on probation at the time the videos were taken and had been ordered to see Levin twice a month. The man said he had told authorities about previous assaults and no one believed him, so he bought a spy camera and brought it to his appointments.
Levin, who immigrated to Canada from South Africa, was frequently used by the courts to assess people and provide expert opinions at hearings.
He served briefly as director for the Regional Psychiatric Centre in Saskatoon and was licensed in 1998 to practise psychiatry in Alberta.
Levin is no stranger to controversy over his work. He faced heated accusations about his time as a military psychiatrist during apartheid in South Africa, where he earned his degree in 1963.
In the 1970s he was a psychiatrist at a military hospital where aversion therapy through electric shocks was allegedly used in an attempt to change the sexuality of gay soldiers. Levin is mentioned in a report entitled the aVersion Project that aimed to shed light on abuses of gays and lesbians in the military by health workers.
Follow @BillGraveland on Twitter