"Those aren't the jobs I'm interested in," Tory said at a press conference at city hall on Tuesday.
He says while bringing in more gambling will require hiring, he is more interested in periphery jobs that come with increased economic interest in the area.
His comments come as the executive committee debates allowing more gambling at the entertainment complex in the northwest part of the city. Tory calls the casino a "catalyst" for other jobs in the area.
Woodbine already has gambling, including horse racing and slot machines. The proposal, which would have to get approval from council next month, would see gambling grow.
Tory says while he is reluctant to diminish casino jobs, he wants to see more convention centre and hotel work.
More gambling at Woodbine would serve to "attract jobs to a part of the city that desperately needs jobs," said Tory.
Tory said there are three reasons he wants an expanded casino at Woodbine, and repeated the word jobs three times.
Other councillors and critics have concerns about mental health issues like addiction and the quality of the jobs the casino would provide.
Officials at Woodbine say they need to expand beyond their core product, which is horse racing.
Jim Lawson, Woodbine's CEO, has said expansion could double the number of visitors to as many as 12 million a year, from 6 million.