Matt Anderson added 15 points in the 25-23, 25-16, 25-20 win for the defending Olympic champions. The Germans were led by Georg Grozer with 12.
The U.S. held Germany at bay for most over the first set, but the Germans went ahead 22-21 when Donald Suxho's return caromed out. The Americans quickly regained the lead and scored the final three points, capped by Matt Anderson's kill.
Germany kept up in the second set until Reid Priddy's ace put the U.S. up 14-10. Anderson stretched the lead to 20-14 with another ace, clocked at more than 73 mph, and Priddy's kill finished it.
The U.S. rode the momentum through the third set, taking the match on Grozer's serve into the net.
The United States opened preliminary pool play with a three-set sweep of Serbia on Sunday. The men's and the women's volleyball teams play on alternating days at London's Earls Court.
The U.S. went undefeated at the Beijing Games and beat favourite Brazil in an emotionally charged final. That team was coached by Hugh McCutcheon, whose father-in-law was stabbed to death at a Chinese tourist site just before the opening ceremony.
McCutcheon took over the U.S. women's team following Beijing, and Alan Knipe took a leave of absence as coach of Long Beach State to lead the men.
The U.S. is currently ranked No. 5 by volleyball's governing body. The team is in a difficult pool in London, joining top-ranked Brazil, perennial powerhouse Russia, Serbia, Germany and Tunisia. In the other pool are Italy, Poland, Argentina, Bulgaria, Australia and host Britain.
The top four teams from each group in the preliminary round will advance to the quarterfinals, and the gold medal match in London is set for Aug. 12.
The Americans are not considered among the favourites in London despite a silver-medal finish in the recent FIVB World League tournament and solid outings against Serbia and Germany.
Top-ranked Brazil has struggled of late, and the team failed to make the semifinals in the recent FIVB World League for the first time since 1998. The Brazilians, who swept Tunisia in the Olympic tournament's opening match, face No. 2 Russia in a late match Tuesday at Earls Court.
Germany, ranked 13th in the world, hired coach Vital Heynen in February and didn't qualify for an Olympic berth until one of the last-chance tournaments in June. The Germans finished ninth in Beijing.
Earlier in the day, Bulgaria surprised Poland 3-1. Tsvetan Sokolov had 28 points, including four aces, in ninth-ranked Bulgaria's 25-22, 29-27, 13-25, 25-23 win to go up 2-0 in pool play.
It was the first real upset on the men's side.
Zbigniew Bartman had 21 points for No. 3 Poland, which dropped to 1-1 in the pool after defeating Italy 3-1 in the opener.
Earls Court was filled with red-clad Team Poland fans and festooned with hundreds of Polish flags, many inscribed with the names of different cities.
The fans were quieted when Bulgaria took the first two sets. But Poland rebounded in a big way to dominate the third and deny Bulgaria the sweep.
Bulgaria held off Poland in the final set, winning the mach on Viktor Yosifov's kill.
"Today you have seen a different Bulgaria" Sokolov said.
The Polish players appeared frustrated throughout the match. Poland has been on the upswing since hiring accomplished Italian coach Andrea Anastasi early last year. Team Polska won the recent World League, beating top-ranked Brazil three times in the early round and again in the finals.
After the match Tuesday Anastasi put his hands on his hips and shook his head.
Poland, which won the Olympic gold medal in 1976, finished fifth in Beijing.
The Bulgarian team came to London in disarray. After the team earned an Olympic berth at a qualification tournament in Sofia, the coach and a top player suddenly resigned to protest their country's leadership of the sport.
Coach Radostin Stoychev and spiker Matey Kaziyski have called for the resignation of Bulgarian federation chief Dancho Lazarov.
"I am really proud that in this moment of chaos, with coaches coming and going, we're showing our character. This is something which fits very well to the nature and spirit of Bulgarian people" team captain Vladimir Nikolov said.
Bulgaria was also involved in controversy in Beijing, when team captain Plamen Konstantinov suddenly went home for a drug test, spurring rumours about possible doping. He missed three matches.
In other matches on Tuesday, Serbia defeated Tunisia 3-1 (25-15, 25-21, 20-25, 25-18), and Italy downed Argentina 3-1 (25-17-21-25 25-17 25-21).