The three-alarm fire started on the 9th floor of the assisted living facility at 80-84 Carlton Street.
Neighbours said the man who died was a quadripeligic who lived on his own.
Toronto paramedics said the victim, who was reported to be in his 40s, died inside a unit of the building located at 84 Carlton St.
The fire started shortly after 9:30 p.m. and was quickly contained
The building, known as Tobias House, has many residents who are wheelchair users, as well as others who need assistance.
There is no word on what caused the fire.
The Ontario Fire Marshal's Office is investigating.