Many of the Chinese players burst into tears as the Japanese celebrated at Earls Court.
The fifth-ranked Japanese won 28-26, 23-25, 25-23, 23-25, 18-16 and will next play the winner of the quarterfinal match between Russia and Brazil.
Saori Kimura and Yukiko Ebata each scored 33 points for Japan, whose finesse countered China's power. Hui Ruoqi had 26 points for China, which had beaten Japan in straight sets in all five Olympic matches between the nations since the sport debuted at the 1964 Tokyo Games.
"I'm delighted to win this match, although it was very difficult from the beginning to the end," Japanese captain Erika Araki said. "It was a very close match."
Japan scored four straight points to go up 5-3 in the deciding set and held off China until Wang Yimei made two straight kills to put her team ahead 9-8. Japan regained the lead and won on Hitomi Nakamici's ace after Zeng Chunlei's dig sailed out to set up match point.
"We were relieved at the end of the match that we won," Japan coach Masayoshi Manabe said.
Wang, a 1.90 metre tall wing spiker known for her powerful jumper serve, had 22 points for China, which was ranked third. Coming off surgery on her right ankle, Wang had problems with the injury in the preliminary round, and against Japan it was heavily wrapped.
The energetic Japanese team kept pace with China in the first set, taking a 27-26 advantage on Ebata's spike before Kimura's kill for set point couldn't be stopped by Hui.
China found a rhythm midway through the second, pulling ahead 18-15 on Xu Yunli's ace, then winning it on her kill. Japan won the third set with another Kimura spike.
The fourth set was tight until Wang's kill set up Ma Yunwen's block to force the deciding set.
"Their defence was very good," Chinese middle blocker Yang Junjing said. "We couldn't follow their tempo with their blocking."
The Japanese have won gold twice in volleyball, but their most recent was in 1976. China has won the gold three times, most recently at the 2004 Athens Games. The Chinese won the bronze on home soil at the Beijing Games.
China struggled going into the games, losing four straight matches at the recent FIVB World Grand Prix while resting Wang's ankle. Chinese coach Yu Juemin caused a bit of a stir when he suggested a three-week vegetarian diet was responsible for the team's poor showing in the tournament.
The team eschewed meat while preparing for the Grand Prix away from their usual training facility, because there were concerns about clenbuterol, which is illegally used to bulk up livestock and produce leaner meat.
Clenbuterol is on the World Anti-Doping Agency's list of banned substances as an anabolic agent that builds muscle and burns fat, and athletes who test positive can face bans of up to two years.