Nahomi Kawasumi and Aya Miyama scored first-half goals to stake Japan to a 2-0 lead at half time before Melissa Tancredi replied for Canada in the 54th minute at City of Coventry Stadium.
Canadian coach John Herdman was pleased with his team's effort against a difficult opponent.
"You look at every one of our players tonight, there wasn't one ounce of Canadian left out there," Herdman said. "They matched the world champions for periods, both defensively and in moments of attack."
Canada’s next match is Saturday against South Africa.
They could be without central defender Candace Chapman, who went down with a leg injury late in the Japan match.
The Canadians finish group play next Tuesday against Sweden, the No. 4-ranked team in the world.
It is conceivable Canada could still advance to the next stage winning just one of the three matches. The top two teams in each of the three groups, along with the two best third-place teams, advance to the knockout stage.
In other women's soccer matches Wednesday, the first day of competition at the London Games, the top-ranked United States rallied from a 2-0 deficit after 14 minutes to defeat France 4-2. Sweden handled South Africa 4-1, host Great Britain edged New Zealand 1-0, and Brazil clubbed Cameroon 5-0.
The match between North Korea and Colombia was delayed more than an hour after London organizers mistakenly displayed the South Korean flag on a jumbo screen instead of North Korea's, prompting the North Koreans to refuse to take the field.
North Korea would beat the Colombians 2-0.