Coach Juan Carlos Reyes says Leosmel Ramos, Wilber Turro, Manuel Suarez and Orlando Sotolongo have left the team and travelled to the United States.
The Games rowing venue in St. Catharines, Ont., is only a few kilometres from the U.S. border.
Reyes says Ramos and Turro dropped out before the regatta began on Saturday while Suarez and Sotolongo were in earlier races and then withdrew.
He said they have been in touch with teammates through Facebook.
Two Cuban baseball players, Luis Yander La O and Yadiel Hernandez, previously defected during a tune-up competition in North Carolina the week before the Games.
Joe Kehoskie, a U.S.-based agent who has represented several Cuban baseball players, said the defections are likely driven by the economic conditions inside Cuba rather than the chance to pursue a sports career in the U.S.
Kehoskie said Cuban baseball players and boxers have starred in the U.S., but he doesn't know of any other amateur athletes who have achieved high-level success.
"I would assume that they're just leaving for economic reasons," Kehoskie told CBC News.
"Unlike baseball players … nobody is going and offering a rower or a hurdler or a gymnast a speedboat trip off the island."
"For these guys it's like the old days."
U.S., Cuba relationship thawing
While the U.S. is in the process of normalizing relations with Cuba, including a recent agreement to reopen embassies, Kehoskie said many Cubans are more concerned with leaving the island now than ever. That's because, he said, the U.S. may end its "wet foot/dry foot" policy, which allows any Cuban who steps foot on U.S. soil to remain in the country.
U.S. President Barack Obama vowed to keep the policy earlier this year. The Cuban government opposes it, saying the policy promotes illegal immigration and human trafficking.
Pan Am Games officials say their protocol for dealing with defections includes working with the Games' integrated security unit and, if necessary, the affected country's national Olympic committee.