Financial terms of the deal were not immediately available.
BMO said the sale will help the bank focus its resources on growing its core U.S. operations, including its institutional fixed income business.
The agreement will see about 130 BMO employees move to Piper Jaffray.
Minneapolis-based Piper Jaffray said the acquisition will broaden its institutional sales, trading and underwriting business and strengthen its strategy and analytics capabilities.
The agreement is subject to regulatory approval and closing conditions. It is expected to close in the fourth quarter of this year.