The man and his children cannot be named due to provisions of the Child and Family Services Act.
He admits to smoking marijuana habitually and to sometimes becoming violent in the past, but never with his children, who have special needs.
He said he's gone through counselling and anger management, and that he'll continue to push for the right to see his children.
Man plans hunger strike on Friday at Parliament Hill
The man said by phone Wednesday night that the judge brought up his weight during the ruling, as well as his hatred for women, which the man denies.
He said the judge said it would be a full-time job to lose weight, and another full-time job to care for the two active boys.
He said the ruling also prohibits him from contacting any Children's Aid Society workers for the next two years.
The man said he plans to start a hunger strike on Friday and create a website about his struggles.
"I'm going on a hunger strike to bring attention to not just my situation, but definitely my situation is paramount," he said. "My children need to know the lengths which I am willing to go to at least have a chance at raising them, and I'm going to go for as long as I possibly can. And while I'm on the Hill I'm going to let people know what the situation really is.
"I am absolutely blown away by this decision, and I will not stand for it," he said.
He once weighed 525 pounds and is now 355 pounds, he said.
"I was fighting for a single chance at custody," he said. "I have never been given a chance."