Lt.-Cmdr. Bruno Tremblay, a spokesman for the navy in Halifax, said the spill was reported at 5 a.m. by crew aboard HMCS St. John's.
He said they are still assessing how much fuel leaked and how it happened.
A cleanup crew was dispatched and the harbour master alerted.
“Our focus right now is to clean it up and minimize the impact on the environment," said Tremblay.
Defence Minister Peter MacKay said he's looking into the spill.
"I really can't tell you a lot... I was advised of it and we're looking into the cause of the leak and of course we'll take all measures to ensure that we're preventing such leaks in the future and take responsibility for the environmental concerns any time there's a fuel leak," he said.
The ship is anchored near Dartmouth Cove, directly across from downtown Halifax.
The powerful smell of fuel on the water was still evident several hours after the spill was reported.
This is the second navy ship spill in the harbour in the past two years. There was another accident when the supply ship HMCS Preserver was being refueled in March 2011.