First responders were called to the scene around 10:30 a.m. ET, where they found the man and dog sitting on a small piece of ice in the middle of the canal.
An eyewitness told CBC News the man was out with his two dogs when one of them jumped into the canal, which is still full of ice. Moments later, the man jumped in after the animal.
Jason Hughes was out for a walk with his family when he saw the incident and called 911.
"I was a little worried. Obviously the water below the ice is really cold. That's certainly a dangerous position to be in."
A firefighter team from Pointe-Saint-Charles responded and deployed a specialized aquatic team. The man was taken to hospital with a minor case of hypothermia, but both the man and his dog are expected to survive.
Hughes said the entire incident could have been avoided if the dog had been wearing a leash.
"My first thought was, this is the reason we have leash laws in Montreal," said Hughes. "I was hoping no one else would jump in after him, because the more people in the water, the harder it is to rescue."
Firefighters are using this as an opportunity to remind people not to risk their lives to save animals.
"My advice is don't jump in the water," said Michel Girard, spokesman for the Montreal fire department.