It was the first loss of the London Games for the Americans.
Serbia scored the game's first three goals to take control and then opened up an 8-3 lead by doing the same to start the second half.
With the win, Serbia moved into first place in Group B with seven points and advanced to the quarterfinals. It can wrap up the group's top seed for the knockout stage by beating Romania in its last preliminary-round game on Monday.
The U.S., which has already secured a spot in the quarterfinals, is one point behind the Serbs, and plays three-time defending champion Hungary in its last group game.