The fire began Saturday morning at Distribution Golf FPC Inc., a company that specializes in recycling used golf balls.
City authorities said the evacuations were precautionary.
The residents were allowed to return home after a provincial environmental team determined there was no health risk from the smoke.
Thick, black smoke poured from the factory in Beloeil and reduced visibility to near zero along nearby Highway 20.
Holiday travellers experienced major delays after the highway was closed in both directions for several hours.
About 70 firefighters were the fire. Two were slightly injured.
One of about 20 employees who were in the building was taken to hospital for a minor health problem.
The cause of the fire is being investigated.