Dennis powered along the 14-kilometre (8.6-mile) flat and windy urban circuit through cycling-mad Utrecht in 14 minutes, 56 seconds.
"I didn't expect to go that fast," the Australian rider said. "I was just thinking '16 minutes, 16 minutes' from what I did in training, and in the end I got a bonus."
His average speed of 55.446 kph (34.4 mph) beat the record by British rider Chris Boardman on the 1994 Tour prologue, albeit over a distance twice as short.
"There's still emotions going on," said a thrilled Dennis, who previously held the world hour record when covering a distance of 52.49 kilometres in February. "The team has done a lot of hard work and it's paid off. Recons, specific trainings, it's all come together."
The last Australian to wear the yellow jersey was Simon Gerrans in 2013.
"I don't want to be a one-hit wonder. I want to hold it for at least one more day," said Dennis, who took silver in the team pursuit at the 2012 Olympics.
His performance in sweltering heat that hit 36 degrees (97F) was even more impressive given that he beat two time trial world champions.
"I think being Australian and growing up in this heat may have been a bit of an advantage for sure," he said. "My specific training between 2-5 p.m. gets you used to that. I have been training in 40 (104F) degrees to get adapted to those conditions and that was the plan of the team ... not to come in blind."
Three-time world champ Tony Martin trailed five seconds behind in second place, and four-time world champ Fabian Cancellara was six seconds back in third.
"I wanted to win. Any other result is a bad one," Martin said. "I feel that I couldn't handle the heat, especially in the second half where I felt weaker."
Nibali, who tried to counter the searing heat by warming up with ice cubes strapped to his back, finished 43 seconds behind Dennis in 22nd place.
"I felt good, and I'm satisfied I gained time on the other top favourites," Nibali said.
He was seven seconds ahead of 2013 Tour champ Chris Froome (39th), 15 seconds up on two-time Tour winner Alberto Contador (46th), and 18 seconds ahead of Nairo Quintana (57th).
"It was pretty tough out there for such a short course," Froome said. "The longer efforts are where my training is geared towards. That was hard but I'm happy to have it out of the way."
Svein Tuft of Langley, B.C., finished the first stage in 25th place while Victoria's Ryder Hesjedal was 87th.
After his ride, Dennis climbed off his bike and watched with a nervous smile from the finish line as his rivals tried and failed to beat him.
They included Tom Dumoulin, who looked in great form but crossed the line eight seconds adrift in fourth.
"It was just a waiting game," a relieved Dennis said. "A stressful two hours, maybe too stressful."
The Netherlands' fourth largest city has 320,000 inhabitants, and most of them seemed to be lining the course route, squeezing next to each other behind railings, or leaning out of windows as they roared on each rider.
The only place where there were no fans was when riders went through a tunnel.
The noise level went from loud to deafening when Dumoulin — a Dutchman with a rather French sounding name — put his head down and pounded his pedals.
After Dumoulin tried and failed, it was Cancellara's turn.
Cancellara used his slick handling skills to attack the corners, but the heat also took its toll on the Swiss rider, who crossed the finish line gasping for air.
Meanwhile, Dutchman Lars Boom started the race despite questions raised over his testing results, and he finished in 23rd.
Pre-race tests on the Astana rider showed a low cortisol level, which can indicate cortisone doping but is not conclusive proof of doping.
"It is important to apply the rules accurately," UCI president Brian Cookson said. "No rules of the UCI or the World Anti-Doping Agency have been broken so he's allowed to ride the Tour de France."
The race stays in the Netherlands for Sunday's second stage, which starts out from Utrecht before ending 166 kilometres (103 miles) later at the heart of the Zeeland Delta.
Cycling along the Dutch coastline promises to be tricky for riders, with heavy gusts and sprays.
The Tour swings into nearby Belgium on Monday.
AP Sports Writer Samuel Petrequin in Utrecht, Netherlands contributed to this report.Suggest a correction