Phelps leaves the sport with a record 18 golds and 22 medals overall. At these games he won four golds and two silvers.
In the final swimming event of the eight-day meet, the Americans clocked 3 minutes, 29.35 seconds.
Matt Grevers led off in the backstroke leg, Brendan Hansen swam the breaststroke, Phelps did his usual butterfly leg and Nathan Adrian was the anchor in freestyle.
The Americans trailed when Phelps dove in, but he pushed them in front.
Japan touched in 3:31.26 to take the silver medal and Australia finished in 3:31.58 to take bronze.