The court gave the institution 90 days to determine whether it has identified all relevant documents sought by Canadian Press reporter Jim Bronskill, who originally requested the Douglas files in 2005 under the Access to Information Act.
The oral ruling from the bench came unexpectedly Wednesday after a panel of three judges heard three hours of legal arguments from lawyers for the federal government and The Canadian Press.
The government has maintained it can't release certain information because it could jeopardize the lives of confidential informants and give away secrets of the spy trade.
But the government offered one concession in Thursday's hearing when Justice Department lawyer Alain Prefontaine said the government now considers the historical value of the Douglas material to be a factor in deciding whether it should be made public.
The court took note of the government's modified position as it granted its appeal of the ruling that The Canadian Press successfully obtained last year for fuller disclosure of the file.
In doing so, the Federal Court of Appeal also tweaked some of the language of the ruling of Federal Court Justice Simon Noel.
"The court agreed that the order that was issued needed to be changed," Prefontaine said in an interview. "To do that, they needed to allow the appeal. If you don't allow the appeal, then you can't change the order."
Prefontaine said he expected Wednesday's order could lead to a resolution of the long-running case sometime after the 90-day review period.
Paul Champ, Bronskill's lawyer, said he wasn't sure what message the appeal court was trying to send Wednesday.
"This is a pyrrhic victory for the government. And frankly, with great respect for the court, I'm not sure what effect the ruling is going to have on this request, or what it was meant to have," said Champ.
"The simple fact is that when Jim Bronskill made his request to Library and Archives Canada he got less than 25 per cent of the documents that they had. As a result of this application to the court, he now has 85 per cent."
Library and Archives Canada consulted with Canada's spy agency, CSIS, when the original request for records was made.
CSIS replaced the RCMP's now defunct security branch when it was disbanded in the early 1980s.
Champ accused CSIS of working to block the release of the Douglas files, even though it turned them over to the archives two decades ago, and considers them to be of no operational value.
Files released so far showed the RCMP's security branch spied on Douglas from the late 1930s to shortly before his death in 1986.
Library and Archives initially released just over 400 pages to Bronskill, and followed that with another 300 pages shortly before the case was first heard in court last year.
The Mounties attended Douglas' speeches, analysed his writings and eavesdropped on private conversations. They were particularly interested Douglas' links to the peace movement and the Communist party.
"CSIS has been following this case very closely," Champ told Wednesday's hearing, which drew about a dozen spectators.
"I'm sure some of the people here are from that agency."