The proposal was to tear down an existing house, divide the property and build two large homes, back to back.
Ted Nixon and some other neighbours fought against the proposal.
"So we will have not only one monster home looking into our backyard, but we'll have a second monster home looking into our backyard. Our privacy is gone," said Nixon.
It's expected the homes will be built by the end of the year.
But many are surprised construction is underway at all.
The original application to build the homes was rejected by city planners. They said it didn't comply with existing bylaws.
But the builder appealed to the Ontario Municipal Board, and the OMB overturned the earlier decision.
"We don't understand why did the OMB overturned this decision," said Nixon. "Why are they not enforcing the bylaws?"
Those concerns - and others like them - have prompted NDP MPP Rosario Marchese to introduce a private member's bill to take the OMB out of Toronto's planning and development issues.
"It should be city planners and the City of Toronto — along with the community — that should decide what's good for them and not some appeal body that's unelected and unaccountable," he said.
But the builder says he followed the process and did nothing wrong.
"We're involved in the community in many ways," said Emil Toma, president of M.A. Homes. "And we build a lot in the area and we're always conscientious of neighbours and go overboard to help out neighbours."