Murray will step down on April 30, 2014, a day before senior deputy governor Tiff Macklem is scheduled to leave.
Murray, who has been a deputy governor since in January 2008, has overseen the central bank's analysis of domestic and international economic developments.
"John has provided unwavering, top-notch intellectual leadership to the bank and its staff — myself included — for more than 30 years," governor Stephen Poloz said.
"His work has had a significant impact on central banking, not only in Canada but at the global level, as his colleagues around the world will readily attest."
The central bank said Murray played a significant role in establishing the theoretical foundations of inflation targeting and helping to refine the way in which it is implemented.
The Bank of Canada adjusts its policy interest rate with an aim to keep inflation at the two per cent midpoint of a target range of one to three per cent over the medium term.
Macklem announced earlier this month that he will leave the bank on May 1, 2014, to become dean at the Rotman School of Management at the University of Toronto.
Macklem had been widely considered the leading candidate to succeed Mark Carney as governor but the job went to Poloz, who has was been governor since June.