Vizer remained defiant, sticking to his guns and declaring of the Olympic establishment: "We don't need cardinals of sport. We don't need popes."
The council of the Association of Summer Olympic International Federations decided unanimously to suspend relations with Vizer's SportAccord pending a review and clarification of its role and governance.
The head of track and field's governing body compared Vizer to a "dictator," and the International Paralympic Committee announced that it was withdrawing as an associate member of SportAccord in protest.
The developments came after Vizer blasted the International Olympic Committee and Bach on Monday in a speech at the opening of the SportAccord convention in Sochi. It marked an unprecedented public rebuke within the Olympic movement and exposed a bitter power struggle at the high levels of sport.
With Bach looking on, Vizer accused the IOC of lacking transparency, ignoring the federations, blocking his plans for new multi-sport competitions and wasting money on an Olympic TV channel and opening and closing ceremonies. He described the IOC system as "expired, outdated, wrong, unfair and not at all transparent."
The summer sports association said it considered Vizer's position "not compatible with the role and mission of ASOIF as a major stakeholder of the Olympic movement."
The issue will be discussed at the ASOIF general assembly in Sochi on Wednesday.
ASOIF represents the 28 sports on the program of the Summer Olympics. Among its members is the judo federation, of which Vizer is also the president.
SportAccord represents more than 100 Olympic and non-Olympic federations, including the 28 ASOIF members. Vizer, who is demanding more power for his organization, was re-elected Monday to a four-year term as president.
With his position under fire, Vizer shot back in an interview broadcast Tuesday night on SportAccord's TV network partner, Euronews, again accusing the IOC of looking out for its own interests.
"I expressed the reality of sport today," he said. "The question is to clean up the system and to make it fair for the benefit of sport. Not a system that defends itself and a specific group or leaders or cardinals of sport because we don't need that and sport doesn't need that.
"I don't care if some people are afraid to say that but I say that. We don't need cardinals of sport. We don't need popes. We need fair leaders who are examples for sports through their attitude, behaviour, measures, actions, initiatives, strategies and vision."
Vizer said the IOC has "lots of tentacles in different directions."
"So the question is very simple: Do we want to clean up the story or cover the story?" he said.
Vizer said he was prepared to fight his case.
"It doesn't matter what are the consequences for myself personally," he said. "My voice is the voice for sport, for reality, for fairness in sport. .... What I say is true and nobody can stop that."
IAAF President Lamine Diack said earlier that Vizer resembled "a chief or dictator coming from nowhere" and telling major sports federations what to do.
The IAAF and international shooting federation both withdrew from SportAccord.
ASOIF president Francesco Ricci Bitti said two more federations, World Archery and the International Canoe Federation, had suspended their involvement with SportAccord, but not withdrawn completely.
UCI President Brian Cookson would not rule out the possibility of the cycling federation pulling out.
The International Paralympic Committee withdrew its associate membership late Tuesday.
"The IPC do not agree with the comments made by President Vizer about the IOC during Monday's opening and certainly do not support his views," IPC chief executive officer Xavier Gonzalez said.
Along with the IAAF, at least 14 federation presidents, including FIFA's Sepp Blatter, signed letters of protest at Vizer's comments.
Vizer has been at odds with the IOC for several years, ever since he first took over SportAccord and sought to establish a "United World Championships" every four years for all the sports. The plan has not materialized.
The IOC snubbed Vizer by deciding not to hold an executive board meeting during the SportAccord convention in Sochi. The board has met during all the previous annual conferences going back to 2003. Vizer was also perturbed after the IOC prohibited the two bid cities for the 2022 Winter Games — Beijing and Almaty, Kazakhstan — from making presentations in Sochi.
Wilson reported from London.