The couple, a 34-year-old man and a 27-year-old woman, were back in court Friday morning to set a date for a bail hearing.
Their appearance in court was short, and no additional charges were laid.
They are currently facing four charges each in relation to one of the children for forcible confinement, assault, assault causing bodily harm and failure to provide the necessaries of life.
The Sherbrooke police told CBC News on Monday that it was possible charges involving sexual assault could be filed against the man, although he has not yet been charged.
Three young children were removed from the family's home. One, a five-year-old boy, was found badly malnourished.
The other two children, girls aged four and six, were in better condition.
A fourth child, aged 14, did not live in the family home but was identified by police as being removed from the couple's care.