The two raised roadways were built in the 1970s, designed to be part of a freeway system that was abandoned after public opposition.
The majority of developers, community and business associations that were consulted gave their support to the plan, which would see the viaducts replaced with a waterfront park, condominiums and affordable housing.
"This is the last piece remaining that needs to go," said architect Richard Henriquez, who helped lead the charge to stop freeway construction in Vancouver.
Developer Jon Stovell supports the proposal but said he is concerned Gastown's resurgence will be affected.
"I'd be very unhappy if the result of removing the viaducts was to strip a few parking stalls from Gastown in order to speed water street up," he said.
"Let's keep Gastown slow and neighbourhood-like."
Burnaby Mayor Derek Corrigan said the move would lead to more traffic congestion for those living outside Vancouver.
"Each time decisions are made to further constrict traffic into the urban core, the more difficult it is for us to manage the impact of that traffic within our communities," he said.
Corrigan said he isn't opposed to the idea of the viaducts coming down, if that's what Vancouverites decide.
However, he said, he wants to make sure it's done right and that goods can move freely from the port to other communities.
Wednesday's public consultation follows the delivery of a report on the benefits of removing the viaducts, the possible new street system and the financial implications of the proposal.