John Coates, who has made six trips to Brazil as part of the IOC's co-ordination commission for Rio, said the Brazilians are behind "in many, many ways" and are in worse shape than Greek organizers were in preparing for the 2004 Olympics.
Despite the critical delays, the Australian said there is no backup plan and the games will take place in Rio.
Coates noted that the IOC had taken the unprecedented step of embedding experts in the host city to help the local organizing committee deliver the games.
"The IOC has formed a special task force to try and speed up preparations but the situation is critical on the ground," Coates told an Olympic forum in Sydney, outlining that construction delays are just part of the problem. "The IOC has adopted a more hands-on role. It is unprecedented for the IOC, but there is no plan B. We are going to Rio."
Brazil has also come under fire from football's world governing body, FIFA, for long delays in construction of stadiums and other infrastructure and the overdue delivery of venues for the World Cup, which kicks off in June. Two years out from the 2016 Olympics, the situation on the construction front is just as bleak.
"We have become very concerned. They are not ready in many, many ways," Coates said. "And this is against a city that's got social issues that also have to be addressed; a country that's also trying to deal with the FIFA World Cup coming up in a few months."
Coates said dealing with three levels of government in Brazil made it harder for local organizers than it was for the heavily criticized organizers of the Athens Games, which were also plagued by construction delays and earned a "yellow light" warning from then-IOC President Juan Antonio Samaranch.
"I think this is a worse situation than Athens," Coates said. "In Athens, we were dealing with one government and some city responsibilities. Here, there's three.
"There is bureaucracy, there is little co-ordination between the federal, the state government and the city — which is responsible for a lot of the construction. The flow of funds from the federal government is not happening quickly enough. We think we need to help facilitate that."
Coates, who chairs the IOC co-ordination commission for the 2020 Tokyo Olympics, said he thinks the IOC has put the message across in Brazil.
"We have to make it (the Olympics) happen and that is the IOC approach, you can't walk away from this," he said. "If it comes off — the first (Olympic) Games on the South American continent, in a magical city in so many ways — it'll be a wonderful experience for the athletes."
Work hasn't begun at Deodoro, a complex for eight Olympics sports venues, and the course that will host golf's return to the Olympic program for the first time in more than a century doesn't have grass yet. Water pollution is a big worry for sailing and other sports.
Concern over the delays has been building over time but hit crisis levels at the SportAccord meeting in Belek, Turkey, earlier this month.
In a rare display of unified, open criticism against an Olympic host, 18 sports federations publicly aired concerns over Rio's preparations, with some sports asking about "Plan B" contingencies.
The IOC has decided to send a senior troubleshooter, IOC executive director Gilbert Felli, to Brazil as part of a series of emergency measures to tackle the delays threatening the games. Coates said other experienced, high-level staff will soon be appointed.
Rio Mayor Eduardo Paes fired back last week at the complaints by sports federations, saying they were making too many unnecessary demands.
Paes said the federations were asking for too many "large things" that won't be used by the city after the Olympics.