Murray withstood 17 aces from Karlovic at Ariake Colosseum.
Gael Monfils of France, who was scheduled to be Murray's opponent, withdrew with a right knee injury he aggravated in Beijing last week, and was replaced by lucky loser Karlovic. Monfils is returning to Paris to have the knee examined.
Despite having break points in the first and seventh games, Murray was unable to capitalize.
Playing for the first time since winning the U.S. Open last month, Murray admitted it was not easy against the towering Karlovic.
"Playing against him you know it is going to be uncomfortable," Murray said. "He is a tricky player."
Murray, who bested Karlovic in the first round at Wimbledon, uncorked three consecutive aces in the eighth game to level at 4-4.
With both players unable to break serve, Murray finally prevailed 9-7 in the tiebreak with a forehand passing shot down the line, after saving a set point when he was down 7-6.
Karlovic, who had 12 aces in the set, squandered a chance to go up 6-3 in the tiebreak when he double-faulted.
Murray, who won the gold medal at the London Olympics, took an injury timeout following the first set after appearing to injure his back early in the match.
"It feels fine now," Murray said of the twinge. "After not playing for a while, then to start making sudden, aggressive movements, it can happen. By the middle of the second set I was fine."
The second set was on serve until the 10th game, when Murray went up 0-40 with Karlovic serving. The Croat saved two match points, but sent a forehand long on the third to give Murray the victory.
Murray said his priorities have changed following his first Grand Slam victory.
"I have reset my goals," he said. "Winning a Grand Slam was so important to me that I was often focusing on the next Grand Slam rather than the next match or tournament. That is going to be different now."
In other action, sixth-seeded Milos Raonic of Canada defeated Radek Stepanek of the Czech Republic 6-4, 6-4, while eighth-seeded Kei Nishikori of Japan ousted compatriot Go Soeda 4-6, 6-2, 6-3.
Tommy Robredo of Spain topped Jarkko Nieminen of Finland 6-2, 6-4, while Jeremy Chardy of France downed wild card Yuichi Sugita of Japan 6-1, 7-5. Qualifier Dmitry Tursunov of Russia dispatched Bernard Tomic of Australia 6-4, 7-5.