The 27-year-old Belgian made his move with a few hundred meters remaining and did enough to withstand a late charge from Slovak sprinter Peter Sagan for the biggest achievement of a frustrating career in which he turned professional at age 23.
"It's difficult to believe what happened today, it's fantastic," said Bakelants, who had a knee operation earlier this year. "Today it may be the first and last time I ever wear the yellow jersey."
He won in 3 hours, 43 minutes, 11 seconds, with Sagan and third-place finisher Michal Kwiatkowski one second behind him.
The 97-mile trek started from Bastia and, after four moderate climbs, finished in Ajaccio where Napoleon Bonaparte was born in 1769.
With the finish in sight, Bakelants found himself with five other riders and instinct told him that he may never get a better chance to make a name for himself.
"I felt the others weren't going at 100 percent so I stayed back, but then I saw the peloton were closing in on me," the RadioShack rider said. "With 500 meters to go I had a look and I saw that I was still 100 meters clear of the peloton. I gave everything I had and I made it by one second. But that doesn't matter, I have the yellow jersey."
It has been a difficult career for Bakelants so far.
"I had a lot of bad luck.
Prior to Sunday, his proudest achievement was off the bike—namely a bachelor's degree in bioscience engineering from the university of Leuven in Belgium.
"I think there's more in life than just cycling, But at the moment cycling's in first place." he said. "I think it's going to be a short night tonight, I don't think I'll sleep much.
"My goal was to win a stage but I didn't think it would happen so fast."
German sprinter Marcel Kittel started the day in the lead after winning Saturday's crash-marred first stage, but the rolling hills took their toll and he finished nearly 18 minutes behind in 169th spot.
"It's a difficult stage and I'm a sprinter, that's why I suffer," said Kettel, who retained the sprinter's green jersey. "I had goose bumps when I went up the hill. So many people were screaming my name. But we were expecting to lose it (the yellow jersey)."
The day's last climb up Cote du Salario was much shorter than the other ones but far steeper.
Spaniard Juan Antonio Flecha and Cyrille Gautier attacked up the final ascent, and Tour favorite Chris Froome then launched a surprise attack to go after Gautier when the Frenchman pulled away. But Froome's attack fizzled out and the main pack swallowed him up.
The day after more than a dozen riders crashed, a small white dog ran out into the road some 2. 5 miles from the line and a potentially dangerous situation was narrowly avoided by a matter of seconds.
A bystander started to run after the dog and then changed his mind, and the dog just managed to reach the other side of the road before the marauding pack passed through.
Cavendish was in trouble all day, struggling to keep up as his teammates tried to drag him up the second climb up Col de la Serra.
However, French veteran Thomas Voeckler had a lot in reserve and chased the four early frontrunners.
Lars Boom and Ruben Perez Moreno were soon caught up, leaving just David Veilleux and Blel Kadri at the front.
Voeckler's attack reeled in Veilleux but then fizzled out quickly, leaving Kadri alone in the lead.
French rider Pierre Rolland attacked on the third climb—the day's most difficult, a sinewy category 2 ascent up the Col de Vizzavona—as he chased his third Tour stage win. But the pack accelerated and chased him down.
Tony Martin of the Omega Pharma-Quick Step team recovered from several injuries to take the start. Martin was one of more than a dozen riders to fall Saturday with the German taken to the hospital after losing consciousness and sustaining bruising to his left lung, cuts to his hip, chest, left knee and shoulder, and his back.
He finished 192nd.
Monday's third stage is the last of the Corsican trio and is again hilly, with four moderate climbs dotted along the 145.5-kilometer (90-mile) route from Ajaccio to Calvi.