Marc Lievremont's squad was pushed hard as flyhalf James Arlidge inspired Japan with two tries to get within four points late in the match before the late French resurgence blew the margin out.
"There was a window in the second half where we could have won the rugby game," Japan coach John Kirwan said. "Not only did we show courage in defence in attack, but also we played evenly with France for 70 minutes of the game."
Lievremont was disappointed with his team's performance.
"Above all, I will retain the victory," he said. "I wanted consistency, and obviously in that respect I am not happy."
Two-time finalist France, bidding to win the tournament for the first time, led 25-11 at the break after tries from lock Julien Pierre, flyhalf Francois Trinh-Duc and winger Vincent Clerc.
"I was already not very satisfied at halftime. I felt we lacked ambition right from the start of the match," said Lievremont, who will be replaced by former France winger Philippe Saint-Andre after the tournament. "We were in some fairly straightforward situations when we had the extra numbers that we wasted ... we weren't playing together."
But Japan replied with 10 unanswered points after the break and still threatened the French line.
"We had them under the pump, they were making uncharacteristic errors. Unfortunately we wouldn't come away with that try we needed to go ahead," Kirwan said. "We will be disappointed tomorrow, because I think there was times when we could have done a lot more when you think about the two tries we gave France (in the first half) and the missed opportunities. We look at this game as if we are equals."
Scrumhalf Dimitri Yachvili's penalty gave France some breathing space before locks Lionel Nallet and Pascal Pape soothed frayed nerves with late tries, and substitute scrumhalf Morgan Parra jogged over in the closing seconds.
France won its previous meeting with Japan 51-29 at the 2003 World Cup, a victory also embellished by late tries.
Having opened its campaign four years ago with a 17-12 loss to Argentina — and with fresh memories of a surprise loss to Italy in the Six Nations in March — France was desperate to get onto the front foot early, and Julien Pierre obliged with a try in the fifth minute.
"We felt that there would be a window in the second half and (if we) put them under pressure they might have some memories of the Six Nations and Argentina," Kirwan said. "World Cups are very different for everyone. This is a special tournament and it creates a special type of pressure."
There was little sign of the complications that lay ahead as France crossed again in the 12th when a careless pass from Alridge, intended for centre Kojij Taira, was intercepted by Trinh-Duc for another converted try.
Alridge got Japan on the board after France captain Thierry Dusautoir infringed, then Yachvili responded with a couple of his own kicks to make it 20-3.
Trinh-Duc then needlessly put his own team under pressure when his up-and-under drifted backward, forcing Nallet to concede a penalty and gifting the momentum back to Japan.
But Japan's opening try had a touch of good fortune about it, when Alridge tried to tap ahead into space and the ball hit Trinh-Duc's leg, falling kindly for him to touch down unopposed.
Trinh-Duc made up for his error, setting the scoring movement in motion for Clerc to cross in the right corner for his sixth World Cup try, putting him two shy of winger Christophe Dominici's French record.
Alridge slotted over another kick to reduce the margin to 14 points at the break.
The French had two tries disallowed within minutes at the start the second half, with Harinordoquy and Nallet ruled to be held up.
Japan started causing trouble for the French defence and, despite Estebanez being replaced by David Marty, the inevitable try came when Alridge found a gap down the right flank and darted inside for his second score.
That prompted Lievremont to replace Trinh-Duc with David Skrela and hooker William Servat made way for Dimitri Szarzewski.
"We didn't do the simple tasks well, we couldn't get the ball out and we couldn't move up in defence," Dusautoir said. "So we couldn't get our game going properly. At that point, I had to focus everyone on the fact we had to get back to doing the hard work."
France withstood a ferocious barrage of attacks on its line, with only some thumping tackles from Dusautoir and Nallet preventing the breach as Japan buzzed around the defence with menace and pace.
Instead, the line just about held and Japan had to make do with another penalty from Alridge, trimming the lead to 25-21.
"We came away with three points when we needed to come away with 10," Kirwan said.
The French seemed unable to get out of their own half for 15 long arduous minutes, before Yachvili slotted over from just in front of the posts to give them some breathing space at 28-21.
Japan's battered left-side defence caved in just at the right time for France, as Nallet, Pape and Parra eased over for tries that gave the scoreline a cushion Lievremont's team hardly deserved.
Lievremont said Skrela, who climbed onto the team bus with his right arm in a sling and Estebanez (hip) are in doubt for the next matches. France next plays Canada, Japan faces New Zealand.
France 47 (Julien Pierre, Francois Trinh-Duc, Vincent Clerc, Lionel Nallet, Pascal Pape, Morgan Parra tries; Dmitri Yachvili 4 conversions, 3 penalties), Japan 21 (James Arlidge 2 tries; Arlidge 3 penalties, conversion). HT: 25-11.