In an interview with TV station CP24, Ford's sister Kathy and mother Diane blamed the pressures of the job, his political opponents and relentless media hounding for his problems.
"The pressure that has been put onto him, since Day 1 since he was elected, it is not fair," his mother said.
"It is not right."
His sister denied Ford was a drug addict or alcoholic in need of rehab, and insisted he would stay on as mayor.
"Robbie is not a drug addict," Kathy Ford said.
"I know because I'm a former addict."
His sister did concede he tends to binge drink and make a "fool out of himself."
"When Robbie drinks, I think he just goes full tilt."
But Kathy said the problem pales in comparison to what she called his achievements.
"He has done so much other things than this," Kathy said.
"My heart breaks for my son," Diane Ford said in her living room interview. "It really, really does, because he's been attacked."
She called the growing scandal — which includes a video showing Rob Ford apparently smoking crack cocaine and another released Thursday showing him threatening to kill someone in a profane tirade — unfortunate.
His mother, who said she would always support her son, called Ford's behaviour as mayor unacceptable.
"To err is human but to forgive is divine...but here there is no forgiveness," Diane Ford told CP24's Stephen LeDrew.
"Forgiveness isn't in the eyes of the media right now."
Ford himself has recently been forced to admit to smoking crack cocaine in a drunken stupor and has apologized for other drunken episodes.
Still, his family believes there's no need to send him to rehab, as many of his closes allies have urged.
Despite his faults, Diane Ford said the pressure that has been brought to bear on her son is "not fair."
Some of the hurt is self-inflicted, she said, but there's nothing he won't recover from.
Both women were emphatic that he won't resign.
"Who can say what's going to happen tomorrow," Diane Ford said.
The crisis has been a "rude awakening," they said, but both insisted he is strong enough to get through the scandal and continue on as mayor.
"All the good that he has done, that's all been overlooked," his mother said.
"That's just so hurtful."
Diane Ford did say the mayor has a problem.
"He's got a huge weight problem," she said.