Monfils, who had only beaten Nadal once before in nine meetings, started well with a strong first serve and an ability to chase down nearly every shot.
The 25-year-old Frenchman broke Nadal to go up 4-2 in the first set. Nadal recovered to lead 4-1 in the second set but Monfils stepped up his game. He won 16 of 21 points — including several jump volleys — and broke twice to go 5-4 up before closing out the match when Nadal hit a return wide.
"I just played my game," said Monfils, who swept aside Viktor Troicki of Serbia to reach the semifinals. "I think last night I played a good match and felt very good about it. I knew today if I did it in same spirit against Rafa it would be great because you need to take advantage of every shot you can make. So I just focused on my game and be aggressive. Then today everything worked."
Nadal praised Monfils for his performance, saying he had "fantastic shots with very good backhands, good forehands, his movements were fantastic." But he also blamed his defeat on being too aggressive in the first set and then some poor play — losing two serves in a row — and some bad luck in the second.
"I didn't play the right way tactically in the first set, in my opinion. I wanted to play more aggressive than usual, playing more inside, don't wait," he said. "So to play against him, I know that I cannot play like I did in the first set. I need to wait a little more. I need to play a little bit, you know, slower and wait my chance."
The second-ranked Spaniard said the second set turned when he had the chance to go up 5-3, but hit a forehand into the net to allow Monfils to break and even the set at 4-4. He said he "lost his concentration" after it appeared Monfils was going to challenge the previous shot but then returned it instead.
"He stopped playing, and, yeah, I lost the concentration, because (it) seems like he was trying for a challenge, no, at that moment," he said. "That was a big mistake for me, because with 5-3 on serve, the normal thing is that I hold the serve, and once I do that, I feel confident to win the match."
The match was overshadowed by Federer withdrawing only 30 minutes before his match against Tsonga. The 16-time Grand Slam winner said his back had not improved since beating Andreas Seppi in three sets on Thursday.
Federer, who was looking for his fourth title in Doha, apologized to organizers but said he didn't want to risk further injury ahead of the Australian Open, which starts Jan. 16.
It is only the second time the 30-year-old Swiss has pulled out of a tournament due to injury in his illustrious career. He withdrew from the quarter-finals at the 2008 Paris Masters, also because of a back injury.
"I don't feel a whole lot of improvement for today, and I just don't think it's the right time to risk anything more right now," said Federer, who has tried several treatments. "I still have pain and that's why it was the only right decision, a difficult one for me ... so it's a sad moment for me and for the tournament and for the fans, but health goes first.
"It came throughout two points against (Grega) Zemlja in the beginning, 2-1 in the first set (of the second-round match), and then I wasn't able to serve properly anymore. Had the same thing yesterday. I was really playing with the hand brake on, and I was just trying to manage the situation, really. So it wasn't very easy to deal with."
Federer plans to fly out of Doha on Saturday as scheduled and is hopeful he will recover with rest and further treatment by the end of the week.
"For Australia, I'm optimistic, just because it's not very good but it isn't crazy bad," Federer said. "I have had bad backs in the past. This is definitely not very good, otherwise I would be playing. But I feel without play and the right treatment, I will get through it in the next few days."
The 16th-ranked Monfils is showing signs he may be rediscovering the form that saw him reach a career high of No. 7 last year.
Among the most athletic players on tour with a willingness to dive and slide, the Frenchman's lapses in concentration has often let him down in big matches, such as losing to Nadal at the U.S. Open two years ago despite leading by a set.
Monfils said what was key to Friday's win was never becoming rattled, even when down 4-1 in the second set.
"It was just a break, so I was just trying to adjust the way I played," he said. "I was just thinking it's one break and I know I can break him."
Nadal, for his part, dismissed suggestions that the loss was a setback to his preparations for the Australian Open and his bid to put behind him a year in which was lost to Novak Djokovic six times including the U.S Open and Wimbledon finals and gave up the No. 1 spot to the 24-year-old Serbian.
"I go to Australia very happy about what I did here. For sure I would love to win the tournament, win today, have another match tomorrow," Nadal said. "I am not lying to you, and my feeling is very positive... So seriously, the only negative thing of today is to lose. For the rest of the things, I am satisfied."
Monfils said that Tsonga, one of his good friends, had been a "bit lucky" in getting the walkover and worried about the two having such contrasting styles. Tsonga has won two out of three matches against Monfils.
"It's really always tough to play against Jo because he have a powerful game and kind of game, you know, it's tough to manipulate him from the baseline," he said. "So I need just to focus also on my game and focus on my mind."
Follow Michael Casey on Twitter at https://twitter.com/mcasey1