Treasury Board President Tony Clement says the new system will include annual performance objectives for all employees, mid-year evaluations and annual written assessments.
It will also offer plans for improving performance, with clear timelines for employees to shape up.
Clement says public servants who fail to improve could be fired or demoted.
The new system goes into effect next April.
While the government already assesses performance of senior bureaucrats, there is no government-wide program to review lower-ranking employees.
Clement says it's impossible that every single public servant is doing a first-rate job.
"There's simply no way, I refuse to believe, that virtually every single person that the federal government hires is going to perform to the standards we expect," he said.
"It's impossible. All this needs to change because that isn't right and it isn't fair."
The minister said the system will be fair to employees, while making the system more efficient.
"Performance management is about ensuring good workers have an opportunity to become great workers and dealing effectively with poor performers," he said.
"Either poor performers improve and become productive employees or we will let them go."
Also on HuffPost