The couple, a 34-year-old man and a 27-year-old woman, were arrested Monday morning and are expected to appear in court tomorrow.
Sherbrooke police spokesman René Dubreuil said four children between the ages of four and 14 were taken away from the couple and placed in foster care outside the city.
Dubreuil said the five-year-old was so malnourished that police feared for his life. He said the other children were in better condition.
The man is also expected to face charges of sexual assault, invitation to sexual touching and sexual contact with a minor.
The 14-year-old did not live in the family home.
Dubreuil said the children were discovered after receiving a tip about possible abuse in the home.