Melissa Tancredi scored the lone goal for the No. 7 Canadians, who managed to battle back against the reigning World Cup champions. Japan took a 2-0 lead and into halftime and had the Canadians on their heels for most of the night.
"We knew it was going to be a tough game tonight, they're world champions and world champions for a reason," Herdman said afterward. "They can handle the ball better than any team in the world.
"But it's the first game in the tournament, there's another five to go for us hopefully, and I think we've learned some lessons tonight. . . that when you're 2-nil down against the world champions, you don't roll over."
The game at City of Coventry Stadium marked Canada's first competition of the Games, two days before the opening ceremonies in London.
The Canadians need to finish top-two in their pool — or be one of the two best third-place teams — to advance to the quarter-finals of the 12-team tournament.
On paper at least, the No. 3-ranked Japanese should have posed the toughest test of the preliminary round for the Canadians.
The squad that shocked perennial power United States in a penalty shootout to win the World Cup last summer in Germany lived up to its billing, passing and moving with the speed and precision of a well-oiled machine in front of 14,119 fans, including a couple hundred red-clad Canadians. Japan had 57 per cent of the possession, and outshot Canada 11-4.
"They're quick, they're crafty, they get in those tight little spaces that you don't think they can get into," said defender Lauren Sesselmann, who made a spectacular sliding goal-line save in the second half. "They're an incredible team."
Nahomi Kawasumi and team captain Aya Miyama scored for Japan.
Kawasumi sprinted onto a clever back-heel pass from Shinobu Ohno in the 33rd minute. Miyama scored on a header, leaping to connect on a long cross in the 44th minute to beat Canadian goalkeeper Erin McLeod.
The Canadians knew full well going in that scoring chances would be few and far between.
Tancredi capitalized on the one really decent shot Canada had, connecting on a long cross from Rhian Wilkinson in the 55th minute.
"At 2-0 down, anything could have happened, we could easily have let the game go," said Canada's star striker Christine Sinclair. "But I think in the second half we came out and played pretty well. Gives us some confidence and in terms of goal difference, not a bad result against them."
Sinclair was barely noticeable as Japan's defence virtually marked Canada's leading scorer out of the game.
Japan's Yuki Ogimi looked poised to score with a wide-open shot in the 51st minute, but Sesselmann dove along the goal mouth to boot the ball off the line.
"When you see players pulling their mates out of the mire, last-ditch tackles, that's what this team's got to be about in those games," Herdman said. "The first line's beaten, the second line's beaten, but somebody's going to put their body on the line and make sure the third one's not. And that's what I love about them tonight."
Sesselmann called the last-ditch save instinctual.
"I was just there as backup just in case, kind of just a last minute thing, like 'oh crap,' and just went," she said.
The Canadians played the final five minutes plus injury time with just 10 players after Candace Chapman left the game with an injury. The veteran defender was helped off the field and immediately out the stadium tunnel for medical attention, and she walked to the team bus on crutches after the game.
Canadian Soccer Association officials said Chapman suffered a "leg injury" but had no other immediate details.
Canada plays 61st-ranked South Africa on Saturday, then battles No. 4 Sweden next Tuesday in Newcastle, a crucial game that could determine the Canadians' Olympic fate.
"If you want to progress through this tournament, moral victories aren't going to get us into the next round," said midfielder Diana Matheson. "We're going to need a result against Sweden for sure."
Canada finished eighth in its Olympic debut in Beijing, losing to eventual champion United States 2-1 in overtime in the quarter-finals.
The squad is looking for a little redemption on the international stage after a miserable showing at the World Cup where they went winless through three games, and scored just one goal, en route to being dispatched in the opening round.
Canada arrived at the Games buoyed by some strong results, including gold at the Pan American Games last fall in Mexico.
Canada's first two games are at Coventry Stadium, 150 kilometres northwest of London and normally home to Coventry City Sky Blues. Security was stiff at the stadium, as fans were required to carry the contents of their bags and purses around in large transparent plastic bags.
The Olympic soccer competition kicked off early due to the length of the tournament, and Canada's game was one of six scheduled for Wednesday.