Mark Towhey, one of Ford's closest advisers whose background is in crisis management, refused to explain his sudden departure after more than a year in the position.
"I am no longer the chief of staff," Towhey said as he left the building. "I did not resign."
Ford himself remained silent again Thursday before leaving city hall late in the afternoon.
His office also did little to explain why the mayor was replacing Towhey, who had been his adviser when he ran for the office.
"Effective immediately, Mark Towhey is no longer working in the office of the mayor," the office said in a brief statement.
"Mr. Towhey has been a intricate part of the mayor's office and has made many valuable contributions."
His office thanked Towhey, and said his deputy, Earl Provost, would take on the role in an acting capacity.
Provost was Ford's deputy campaign manager and became deputy chief of staff in August last year.
Last week, both the American-based website Gawker.com and the Toronto Star reported they had seen — but not obtained — a video showing Ford appearing to smoke crack cocaine. They said it was made by a west-end drug dealer who was shopping it around for six figures.
Neither of the reports about the video has been independently verified and the Star itself said it could not vouch for its authenticity.
Gawker has been trying to raise $200,000 to buy and post the video, reaching close to $160,000 by late Thursday.
The website did not respond to a request for comment on its "Crackstarter" campaign.
However, in a note posted on the site, Gawker editor John Cook said his confidence in completing a deal to buy the video has "diminished" because the dealer who apparently has it has been incommunicado in recent days.
"The owner of the video is presumably frightened and skittish, and it's not entirely unreasonable that he would go to ground," Cook said.
"At this point, we have no idea why he is out of touch or if he even knows about the 'Crackstarter' campaign."
Despite calls from friends and rivals to confront the allegations head on, Ford has said little about them beyond calling them "ridiculous" and suggesting the Star was out to get him.
Towhey said his departure did not come as a shock.
"The mayor and I spoke about it this afternoon," Towhey said as reporters trailed him through the underground parking lot.
He refused to elaborate on the conversation or say what he had urged Ford to do about the alleged cellphone video, but did say it was up to Ford to take or leave the advice.
"My conversations with the mayor are between the mayor and me," Towhey said.
"My advice to the mayor is my advice to the mayor."
According to his LinkedIn profile, Towhey's background is in crisis management.
He formed Towhey Consulting Group and was at the helm for about 12 years before becoming Ford's director of policy and strategic planning following the mayor's election in 2010.
His departure comes a day after the Toronto District Catholic School Board announced it had dropped Ford as volunteer coach of his high school football team, the Don Bosco Eagles.
Ford, who allegedly referred to the players disparagingly in the video, has long cited the team as an example of his selfless dedication to others.
The Catholic board made no reference to the crack cocaine allegations, saying only it had decided a new direction was needed.
However, Coun. Jaye Robinson said Thursday she was told Towhey's departure was related to Ford's football coaching, which had prompted criticism the mayor was devoting too much time to the volunteer job.
"It was a tipping point issue," Robinson said.
Ford's biggest supporter, his brother Coun. Doug Ford, said Thursday he would not discuss "personnel issues" but said Towhey's leaving was unrelated to the crack scandal.
On Wednesday, the councillor gave a statement in which he lashed out at the media over the allegations that have dogged his brother.
"Rob is telling me these stories are untrue, that these accusations are ridiculous," Ford said. "And I believe him."
The councillor said his brother has stayed silent on advice of his family and lawyers.
Robinson said Towhey had told her about his displeasure that Doug Ford had held the news conference to deny the cocaine allegations and that the mayor had not done so himself.
"Mark Towhey did have a conversation with Doug Ford to encourage him not to proceed with the media conference, but Doug, as we all know, spoke to the media," she said.
Towhey is a former captain with the Canadian Forces, where he spent 14 years.
Also on HuffPost