The building had housed an apparel store that had operated at the corner for more than 100 years before the building was sold last fall.
Despite its stone carvings, the building does not have heritage designation under the city bylaws. The new owner was granted a demolition permit on Friday and over weekend the work began.
Architect Catherine Nasmith, president of the Toronto branch of Ontario's Architectural Conservancy, said the building should have been preserved.
"It's clear to anybody looking at that building that it had heritage value," she said Monday during an interview on CBC Radio's Metro Morning. "The carvings on it were wonderful. It had been a business that had been in the city for 100 years."
"What you're seeing there is a city that's so far behind in identifying its heritage buildings that this kind of stuff can happen."
On Twitter, the city's chief planner Jennifer Keesmaat suggested the developer worked quickly on purpose.
"This developer has acted rashly precisely b/c he feared it would have *heritage* value, as TBD the city," she wrote.
Developer Sam Mizrahi bought the building in October. He told the Toronto Star he doesn't believe the building has heritage value.
Nasmith said many developers don't want heritage designation on buildings they own because it can limit options for developing the property.
"The development industry doesn't like dealing with heritage, they want to get rid of it," she said.
She said part of the problem is that it can take years for heritage designations to be reviewed.
"It takes the city ages to put any of this stuff into place," she said. "Once it's damaged and torn down, there's nothing you can do about it."
Nasmith said she wants the processed changed so that properties that are subject to demolition applications are first investigated for possible heritage value.