The man was walking along Cooper Street just west of Kent Street when the chunk of ice fell from a building above just before 5:15 p.m. ET, paramedic spokesman J.P. Trottier told CBC News.
"It was a huge chunk of ice," Trottier said. "It did hit him head-first and in the shoulder."
The man was conscious when taken to hospital in serious but stable condition, paramedics said.
Witness Richard Latour told CBC News that the man "looked rattled" as he was ushered into an ambulance.
He said that another "massive piece of ice" smashed into a parked car at a neighbouring home moments later.
"Probably wise to tell people to watch for falling ice because it has been wild today [Tuesday], at least on Cooper," he said.
After an extremely cold February in Ottawa, the temperature rose to a high of 8 C Tuesday afternoon, making it the warmest day so far this year.