That's the key recommendation of the government's red tape reduction commission.
"We are recommending the adoption of the one-for-one rule," said the group's chairman, Maxime Bernier, the minister of state for small business.
"So every time the government proposes a new regulation, it must eliminate an existing one. Applying the one-for-one rule, or any other goal, for controlling regulation and its burden on businesses requires a rigorous standard."
The group, which reviewed the burden on businesses of complying with regulations, released its final report after consultations and roundtables across Canada.
The red-tape commission was made up of six Conservative MPs and six business people.
The group says the auditor general should be put in charge of monitoring efforts to cut red tape.
"Adding red tape oversight to the mandate of the auditor general is a really innovative idea," Catherine Swift, head of the Canadian Federation of Independent Business and a member of the commission, said in a release.
"If the federal government moves on this, it will show it is very, very serious about eliminating dumb and ineffective rules and improving government customer service."
Swift said paperwork caused by government regulations costs businesses more than $30 billion a year. She said the commission's recommendations could save companies as much as $7.5 billion annually.
The commission also recommends that the government reduce the amount of information it demands from businesses and that it allow more information to be filed electronically to reduce costs.
It also proposes bonuses for senior bureaucrats who find ways to cut red tape.
The commission's report was given to Treasury Board President Tony Clement, who noted the number of regulations at the federal level.
"There are 2,600 regulations in Canada for which the federal government is responsible," he said.
"Now, some of these regulations have to do with safeguarding the products we buy, or the services we receive, or even the quality of our air and water, and that's a good motivation.
"And yet, while some regulations generally aim to protect Canadians, others may have adverse effects on the economy, job creation and growth."
Clement and Bernier acknowledged that provincial and municipal regulations can still be a burden on business, but said cutting red tape at the federal level may eliminate duplication with other jurisdictions.
Last week, The Canadian Press reported that documents obtained under the Access to Information Act showed MPs on the commission racked up close to $60,000 in travel and hospitality expenses attending meetings in different cities.
Nearly all those expenses were incurred between last January and March.
Expense forms showed New Brunswick Conservative MP Rob Moore and his staff accounted for well over half the travel and hospitality costs incurred by MPs on the commission.
A Treasury Board spokeswoman acknowledged cheaper travel could have been arranged.
"We agree that lower cost options for travel should have been found in some instances, and are committed to reducing those going forward," spokeswoman Jenn Gearey said at the time.
By Steve Rennie
Also on HuffPost