Oliver's 19-month term as finance minister ends Wednesday when prime minister-designate Justin Trudeau names the person who will take over the portfolio.
The former Conservative MP tells The Canadian Press that his final day as finance minister was quiet and that he spent it writing reference letters for his staff.
The incoming Liberals won last month's election after pledging to run deficits of up to $10 billion in each of the next three years so they could boost infrastructure spending as a way to stimulate the economy.
Oliver says he's proud of the Conservative government's balanced-budget approach and he hopes his replacement makes sure the Liberal shortfalls aren't deeper than promised.
He says the defeat is bittersweet but notes the Conservative campaign could not overcome voters' powerful desire for change.
Oliver is the first sitting federal finance minister to go down in defeat since 1993.
Also on HuffPost