Ford, who coaches the team, says he did not issue any orders to the Toronto police or the Toronto Transit Commission to have the buses emptied during afternoon rush hour on Thursday.
The buses were abruptly rerouted after police requested the vehicles when the team's football game ended early.
Ford told reporters Monday that he was assisting police when he made a follow-up call to the TTC when the bus failed to arrive after 20 minutes.
A frustrated Ford said that he was being unfairly blamed for the incident when it "wasn't my call" for the buses to be ordered in the first place.
It's unclear what prompted the call but Ford said police had safety concerns after a confrontation on the field ended the game.
"I did absolutely nothing wrong," said Ford. "What bothers me the most is when you find out people were told to get off the bus. I'm all about service. That drives me nuts."
He accused the media of unfairly targeting him.
"It's terrible that these people got kicked off a bus. That's not my call. I don't know how much more clear I can be but I've said this over and over and over and over again," said Ford.
"You guys are trying to go after me on something. There's nothing here. I'm as clean as the days are long."
The mayor has been accused in the past few weeks of putting responsibilities with his football team over that of city council.
Meanwhile, in an internal email over the weekend, the head of the TTC said he's "not happy" that fare-paying customers were "inconvenienced" during the incident.
CEO Andy Byford said that he had no idea why the buses were used and that he will be following up on the situation with Ford's office and the Toronto police.