Impey became the first South African to wear the leader's jersey in the race's 110-year history on Thursday. He inherited it from one of his teammates after the sunny sixth stage ended in a victory for German sprinter Andre Greipel.
"History made, so it's one of South Africa's proudest moments. I'm sure everybody's going crazy back home," Impey said. "For me to wear the yellow jersey now, I'm sure it will be something special."
In April 2009, at the low-key Tour of Turkey, Impey was leading that race, too. But he made headlines for another reason, when Dutch cyclist Theo Bos grabbed his jersey and threw him head-first into the metal crash barriers.
It was an incredibly dangerous move by Bos, who was banned for only one month as a result.
Impey, however, was bedridden and on a liquid diet for two months with fractures to his jaw and lower back.
Fast forward to July 4, 2013 and Impey was all smiles on the podium, with cycling's most famous jersey on him.
"Hopefully people will stop talking about that now and will talk about the yellow jersey ... It's been a long journey," said Impey, whose early amateur career in Europe was difficult.
After making his mark on the Tour, the Johannesburg-born Impey sent out a goodwill message to the man who changed his country: the 94-year-old anti-apartheid leader and former South African president Nelson Mandela, who is critically ill.
"Nelson Mandela's obviously a fantastic icon, a fantastic man," Impey said. "Sending him best wishes and hopes he gets well soon. He's definitely made our country a better place."
Impey now leads thanks to his Orica Greenedge teammate Simon Gerrans — who had worn the yellow jersey since the Orica squad won Tuesday's dramatic team time trial by less than one second.
"Daryl was a huge part of me getting the jersey so I thought it was a nice gesture to be able to pass it on to him now," Gerrans said. "Hopefully for a few days."
Impey faces a tough challenge on Friday's seventh stage from Montpellier to Albi, which features a four-mile category 2 climb up Col de la Croix de Mounis. It's only the start of a demanding few days in the race.
But he's used to tough times. His career was forged in bad luck — and bad food — before Greenedge gave him his break when they signed him last year.
"It's not often that a team can show such belief in you so soon," he said. "They've always got my back."
The early days of cycling in Europe, riding for the Marseille-based amateur team Velo Club La Pomme, were tough.
"I was only 20 years old so pretty young and probably immature," he said. "It was tough to fly overseas and learn a new language, but it taught me a lot about myself, and the season didn't work out as planned. I learnt (about) eating badly, to trying to get too skinny, to trying everything basically to being a professional bike rider."
Now his task will be trying to try and lead a normal life back home.
That would have been easier before, in a country whose sporting icons hail from cricket and rugby — Mandela presented Francois Pienaar with the trophy when South Africa won the 1995 rugby World Cup— but cycling's 'maillot jaune' means no more anonymity.
"It will definitely change my life. The rugby players are obviously a little bit higher profile because it's one of our major sports," he said. "Hopefully people will start recognizing me, maybe."
He also hopes it will help promote the sport even more.
"(Cycling) didn't get much support from the country itself so guys had to make their own way," he said. "I think it just shows you the talent that we actually do have in South Africa."
Meanwhile, Greipel won Thursday's stage in temperatures hovering around 30 degrees Centigrade (86 Fahrenheit), surging ahead some 500 metres from the line to beat Peter Sagan and Marcel Kittel.
"It was a very nervous stage today," Greipel said.
The 30-year-old Greipel dedicated his fifth career Tour stage win to his Lotto-Belisol teammate Jurgen Van den Broeck, who pulled out before the start after failing to recover from a knee injury sustained in a crash on Wednesday.
"Sometimes happiness and sadness can be so close together," Greipel said. "It shows the character we have in this team."
Mark Cavendish crashed close to the end and pedaled hard to rejoin the main pack, finishing fourth the day after winning his 24th career Tour stage to close the gap on archrival Sagan in the sprinters' green jersey contest.
Greipel's win moved him above Cavendish and into second place, 29 points behind Sagan. Third-place Cavendish is 40 points behind Sagan.
Another win for Cavendish would have tied him with Andre Leducq for third on the all-time list of stage winners. But he lacked his usual powers of acceleration.
"It took a lot of energy to get back in front," he said.