The Pan American Sports Association said three of the athletes tested positive for anabolic steroids.
It said Mexican weightlifter Cinthya Vanessa Dominguez Lara tested postitive for oxandrolone, which is typically prescribed to help people regain weight they lost due to illness or injury.
Nelson Gomez, a baseball player from Puerto Rico, tested positive for boldenone, while another baseball player, Javier Jesus Ortiz Angulo of Colombia, tested positive for stanozolol, the organization said.
Both substances are also given to race horses. Major League Baseball recently launched an investigation into stanozolol use after four players tested positive for the steroid over 16 weeks. It's also the drug that led to star Canadian sprinter Ben Johnson being stripped of his gold medal at the 1988 Seoul Olympics.
The Pan Am Sports Association said the fourth athlete — Mario Mercedes Castillo, a baseball player for the Dominican Republic, tested positive for dimethylbutylamine, a banned stimulant.
The Peruvian delegation also said Friday that swimmer Mauricio Fiol, the silver medallist in the 200-metre butterfly, has also tested positive for steroids and has been suspended from the team.
In a statement, the Peru team said, "We are profoundly sorry for the incident, which affects not only the athlete but all Peruvians."
Fiol said, "I don't understand how this could have happened. I've always promoted fair play."
Fiol has the option of asking for the result of the B sample.
Canada could benefit if Fiol is stripped of his medal as Zack Chertat's bronze would be upgraded to silver while teammate Alec Page would move up to third place.
The Pan Am Sports Association said it will not confirm or comment on any potential doping cases for 48 hours after receiving test results in order to give the affected athletes a chance to address the allegations.
Any further sanctions against the four removed from the Games will be determined by their sports' governing bodies based on the World Anti-Doping Agency's rules.
Officials with the organization said every major sporting event has a number of anti-doping violations and the Pan Am cases so far fall within the normal range.
Still, Eduardo De Rose, chairman of the organization's medical commission, said he was surprised there were so many cases early on in the Games, which opened a week ago.
New rules issued in January by the World Anti-Doping Agency have led to much more rigorous testing, De Rose said.
"The controls are much harder than the controls we had in Guadalajara," which hosted the last Games in 2011, he said.
Earlier this week, the World Anti-Doping Agency urged organizers of the Games and other large multi-sport events to follow the example of the International Olympic Committee and store samples for possible testing in the future.
Samples from Olympic Games are stored for up to 10 years, meaning that an athlete who cheated may be caught a decade later by new testing technology.
The head of WADA said storing samples from events like the Pan Am Games would be a further deterrent to cheaters and remind them that they are being targeted.
— With files from The Associated Press