The French didn't make it easy on themselves, failing to put the game away in the final minute of regulation.
Gomis missed an open lay-up after a fast break and Isabelle Yacoubou missed the second of two free throws.
Trailing 65-62 with 3 seconds left, Australia's Belinda Snell got the rebound and banked in a 3-pointer from just past halfcourt to send the game to overtime and whip the crowd into a frenzy.
"I was very upset, it was heavy on my mind in the last five minutes. If I had scored we would have won the game (in regulation)," Gomis said of her miss.
"We made a tactical mistake, we should have fouled (Snell) but that is basket. That kind of a shot happens once in a thousand attempts. We didn't let it bother us, we knew we still had to play five more minutes and that it just wasn't over," said Gomis, who scored all of her points in the second half.
Australia was the silver medallist the last three Olympics and is ranked No. 2 in the world.
"It's a big, big win. It's not easy at the Olympics to beat Australia, they won medals," said France captain Celine Dumerc, who finished with 13 points.
"It wasn't our target to beat Australia, for us it's better to beat Canada and Britain," she said.
France also won against Brazil and is now 2-0 in the tournament. It can win Group B after beating two teams it wasn't expected to and still to face teams it is expected to beat.
Russia is also 2-0 in the group after beating Brazil 69-59 behind 14 points from Evgeniya Belyakova. Russia was the bronze medallist at the last three Olympics and is the reigning European champion. Canada beat host Britain 73-65 behind Shona Thorburn's 18 points.
Britain has now lost both games in its first Olympics since 1948, while Canada is 1-1.
The United States, seeking its fifth straight gold, rolled past Angola 90-38 in Group A. Candica Parker had 14 points for the Americans.
Australia had to play the extra session without stars Lauren Jackson and Liz Cambage — both had fouled out.
France has been on the rise as a women's basketball power lately. The French won the European Championship in 2009 and qualified for the Olympics for only the third time.
"We played a very bad first half in offence but we looked at the score and we were plus one so we said, 'Common girls, tonight we can beat this team. Maybe not tomorrow, but tonight we can beat them,'" Dumerc said.
The French didn't let themselves be upset by Snell's desperation shot at the end of regulation.
"Amazing, this shot of Belinda. I was laughing because it's her, we played together, I was saying, 'Don't do this now.' We thought the game was over, but it was our responsibility, we should have fouled before.
"But we came back for the extra five minutes, we knew they had some foul trouble and that if we continue to play like this we will win this game," Dumerc said.
Dumerc and Snell played together for Bourges in the French League in 2006 and 2007.
Suzy Batkovic led Australia (1-1) with 17 points and Jackson added 13 before fouling out.
Earlier, Emine Tugba Palazoglu and Bahar Caglar each scored 11 points to help Turkey beat the Czech Republic 61-57 in Group A.
Nevriye Yilmaz also added 11 for Turkey (2-0), which has made the most of its first trip to the Olympics in women's basketball.
Jana Vesela scored 19 to lead the Czech Republic, which won the silver medal at the 2010 world championship but has dropped its first two games at the Olympics.
Also in Group A, Chen Nan scored 28 points and Miao Lijie added 17 to lead China to a 83-58 victory over Croatia.
Chen and Miao played major roles in 2008 to help China finish fourth at the Beijing Games and are major reasons why China has opened this competition with victories in its first two games. China beat the Czech Republic 66-57 in its opener.
Sandra Mandir scored 22 points to lead Croatia, which is still winless after two games in its first Olympics.Suggest a correction