Mutai stepped up the pace in the last third of the race but Kimetto kept up as the other front runners dropped out after the 32nd kilometre (20th mile).
The two were on world record pace but slowed in the final two kilometres to miss Patrick Makau's mark of 2 hours, 3 minutes, 38 seconds set in Berlin last year.
Mutai finished in 2:04:15, the fastest time of the year and the sixth-fastest ever.
"We slowed at the end. I tried to push but I saw that I could not do it because I had a problem with one of my legs," Mutai said.
Kimetto was only one second behind as the two ran through the landmark Brandenburg Gate in mild sunshine to cross the finish line.
He was the fastest runner-up and the fastest finisher in a debut marathon, apart from Boston, where the up-and-down course is not certified for records.
Another newcomer, 19-year-old Geoffrey Kipsang, completed the Kenyan top-three and finished in 2:06:12 in his first marathon.
Mutai already has the fastest marathon time ever recorded, when he ran 2:03:02 in winning the Boston race in 2011. Because of Boston's course, that time is not recognized as the world record. Excluding Boston, Mutai and Kimetto had the fourth and fifth fastest times in history.
Mutai also won New York last year. But this year had been anything but good for Mutai — until Sunday.
He struggled in hot conditions and had to abandon the Boston race, one of the factors that prompted Kenyan officials to drop him from the Olympic team. The London gold medal went to Stephen Kiprotich of Uganda.
Neither Kiprotich nor Makau competed in Berlin this year.
Aberu Kebede of Ethiopia won the women's race in 2:20:30, ahead of teammate Tirfi Tsegaye, who clocked 2:21:19. Olena Shurhno of Ukraine was third in 2:23:32.