The 27-year-old Belgian made his move with a few hundred meters remaining and the RadioShack rider did enough to withstand a late charge from Slovak sprinter Peter Sagan for the biggest achievement of a frustrating career that only saw him turn professional at the age of 23.
"It's difficult to believe what happened today, it's fantastic," said Bakelants, who had a knee operation earlier this year. "Today it may be the first and last time I ever wear the yellow jersey."
He won in three hours 43 minutes 11 seconds, with Sagan and third-place finisher Michal Kwiatkowski one second behind him. In the overall standings, Bakelants is one second ahead of veteran British rider David Millar.
Victoria's Ryder Hesjedal finished in 40th a day after crashing in the opening stage.
The 156-kilometre trek started from Bastia and, after four moderate climbs, finished in Ajaccio where French emperor and military mastermind Napoleon Bonaparte was born in 1769.
With the finish in sight, Bakelants found himself with five other riders and instinct told him that he may never get a better chance to make a name for himself.
"I felt the others weren't going at 100 per cent so I stayed back, but then I saw the peloton were closing in on me," he said. "With 500 metres to go I had a look and I saw that I was still 100 metres clear of the peloton. I gave everything I had and I made it by one second. But that doesn't matter, I have the yellow jersey."
It has been a difficult career for Bakelants so far.
"I had a lot of bad luck. I've had two operations. I fell at the Tour of Lombardy in 2010, I fractured my right knee and left elbow. You know, things like that take time to heal," he said. "This year I had bad luck as well, an operation on my right leg. I worked very hard to come back."
Prior to Sunday, his proudest achievement was off the bike — namely a bachelor's degree in bioscience engineering from the university of Leuven in Belgium.
"I think there's more in life than just cycling," he said. "But at the moment cycling's in first place."
German sprinter Marcel Kittel started the day in the lead after winning Saturday's crash-marred first stage, but the rolling hills took their toll and he finished nearly 18 minutes behind in 169th spot.
"It's a difficult stage and I'm a sprinter, that's why I suffer," said Kettel, who retained the sprinter's green jersey. "I had goose bumps when I went up the hill. So many people were screaming my name. But we were expecting to lose it (the yellow jersey)."
The day's last climb up Cote du Salario was much shorter than the other ones but far steeper.
By the time the pack reached the foot of it, Kittel and British sprinter Cavendish were among a small band of strugglers drifting further and further away.
Spaniard Juan Antonio Flecha and Cyrille Gautier attacked up the final ascent, and Tour favourite Chris Froome then launched a surprise attack to go after Gautier when the Frenchman pulled away. But Froome's attack fizzled out and the main pack swallowed him up.
"I thought it might be a good time, just to push on a little bit," Froome said. "It's always good to keep people on their toes."
Although he did not lose any time to his rivals, two-time former champion Alberto Contador felt the after-effects of his crash on Saturday, when his left shoulder was grazed.
"There is pain in your whole body," the Spaniard said. "I'm hoping to be better tomorrow."
The day after more than a dozen riders crashed, a small white dog ran out into the road some 4 kilometres and a potentially dangerous situation was narrowly avoided by a matter of seconds.
A bystander started to run after the dog and then changed his mind, and the dog just managed to reach the other side of the road before the marauding pack passed through.
Cavendish was in trouble all day, struggling to keep up as his teammates tried to drag him up the second climb up Col de la Serra.
However, French veteran Thomas Voeckler had a lot in reserve and chased the four early frontrunners.
Lars Boom and Ruben Perez Moreno were soon caught up, leaving just Canadian David Veilleux and Blel Kadri at the front.
Voeckler's attack reeled in Veilleux, who hails from Cap-Rouge, Que., but then fizzled out quickly, leaving Kadri alone in the lead.
Veilleux finished in 116th.
Chasing his third career Tour stage win, French rider Pierre Rolland attacked on the third climb — the day's most difficult, a sinewy category 2 ascent up the Col de Vizzavona. But the pack accelerated and chased him down.
Svein Tuft of Langley, B.C., is in 182nd.
Monday's third stage is the last of the Corsican trio and is again hilly, with four moderate climbs dotted along the 145.5-kilometre route from Ajaccio to Calvi.