In Beijing, there was a case of pre-games mumps and a slow start to overcome en route to his third straight gold medal. Four years earlier at Athens, he lost a protest early in the regatta and then climbed all the way back from 19th to the top step on the podium.
Now, sailing in home waters, he needs to find a way to beat surprising Danish sailor Jonas Hoegh-Christensen.
Yes, there are still six races left, plus the medals races for the top 10.
But this is Ben Ainslie, and no one thought he'd be sitting third after four races, especially sailing in home waters.
Hoegh-Christensen, a smiling, unassuming 31-year-old, has beaten Ainslie in each of the first four races and remained in first place overall on an increasingly nasty Monday afternoon on Weymouth Bay on the English Channel.
Hoegh-Christensen had finishes of second and seventh to keep the overall lead with 11 points. Ainslie finished 6-12 to slip from second to third place with 22 points, behind Jonathan Lobert of France with 19.
"It was a tough day. It wasn't very good at all. I'm frustrated with how I sailed," said Ainslie, who's trying for his fourth straight gold medal and fifth straight Olympic medal overall.
If he wins gold, the 35-year-old Ainslie would surpass Denmark's Paul Elvstrom as the greatest sailor in Olympic history. Elvstrom won four straight golds from 1948-60.
After Hoegh-Christensen led at every mark to win the first two races Sunday, Ainslie joked that the Dane "was on a hotline to Paul Elvstrom."
"Yeah, I think I need to get on the phone to Denmark as well and get some of that," Ainslie said Monday. "Everyone's having ups and downs. Jonas did in the second race. But as far as my own performance goes, I wasn't happy with that at all. I have to improve things for the rest of the week."
Ainslie has set the bar so high that even being in third place has raised eyebrows.
"It will get me fired up for the rest of the week," he said.
Races 5 and 6 are scheduled for Tuesday, when the weather is expected to worsen.
After these games, Ainslie will sail with Ben Ainslie Racing in the second season of the America's Cup World Series. He'll then join defending champion Oracle Racing for the 34th America's Cup on San Francisco Bay in 2013, with eyes on leading a British challenge for sailing's marquee prize in the 35th edition.
Hoegh-Christensen will return to his job with concert promoter Live Nation. After finishing sixth in the Beijing Olympics, he quit sailing for three years.
While he came in well-regarded, no one expected this.
"I focused on my own race today and again had a good day," he said.
He wasn't as perfect as he was on Sunday. In the start of Monday's second race, he hit the pin-end boat and had to do a 360-degree penalty turn. He was 21st out 24 boats at the weather mark. In an Ainslie-esque move, he worked his way up to seventh.
"It was just a big fight to get back. Got back to seventh. Very happy," he said.
"I didn't sail a very good race," he added. "I was out of phase a couple times with the shifts, and the start was not very good, but I pushed really hard and really gave my all to get back. I got a bit angry with myself for the mistakes I made. You've got to push a bit extra."
Sweden's Daniel Birgmark won Race 4. New Zealand's Dan Slater won Race 3, with Hoegh-Christensen taking second.
There are 10 races, with every sailor discarding his worst finish. The top 10 advance to the medals race on Sunday.
In the venerable Star class, defending gold medallists Iain Percy and Andrew Simpson finished 3-2 to jump into first with 18 points. Brazil's Robert Scheidt and Bruno Prada, who took silver behind the Brits in 2008, went 9-6 and dropped to fourth with 20 points. Americans Mark Mendelblatt and Brian Fatih went 5-3 to move into sixth with 27 points.
Another America's Cup sailor, Australia's Tom Slingsby, finished second and first to take the lead in the Laser class. He sails for Oracle Racing. American Rob Crane went 35-42 and is 39th place.
Ireland's Annalise Murphy won both races in the women's Laser Radial.
The 49er skiffs, the fastest boats in the Olympics, made their colorful regatta debut.
Denmark's Allan Norregaard and Peter Lang were in first place with six points after the first two of 15 races. Jonas von Geijer and Niclas During of Sweden were second with eight points, followed by Australian favourites Nathan Outteridge and Iain Jensen with nine. Outteridge is skipper for Team Korea in the America's Cup World Series.
As the 49ers round the windward mark, the sailors hoist their national flag gennakers.
The windward mark was just off the Nothe Fort, where thousands of paying fans sat on a grassy slope.
"The fans, you can hear them from hundreds of yards away. It's a thrill, especially raising up the American flag for a kite," Moore said. "It's something special."