Juan Carlos Reyes said Liosmel Ramos and Wilber Turro dropped out before the regatta began on Saturday, while Manuel Suarez and Orlando Sotolongo competed in earlier races and then withdrew.
The four have been in touch with teammates through Facebook, Reyes said.
The Games rowing venue in St. Catharines, Ont., is only a few kilometres from the U.S. border.
Games officials said Wednesday morning that they weren't aware of the defections.
"I'm not aware of the specific point but we have a protocol that we would follow. This is something that is dealt with through our integrated security unit and if necessary, through the relevant national Olympic committee," CEO Saad Rafi said.
Canadian immigration officials said they couldn't comment on asylum claims south of the border, while U.S. Customs and Border Protection said they couldn't release information on specific cases due to privacy concerns.
"We continue to seek to promote safe, legal and orderly migration from Cuba under our migration accords and deter dangerous unlawful migration from Cuba," the department said in a statement.
"Generally, if a Cuban national arriving at a CBP port of entry or between ports expresses fear of return to Cuba and/or their country of last residence, he or she is inspected and paroled. CBP officers and agents will verify whether the individual has any criminal or existing U.S. immigration history," it said.
Cuban refugees can apply for permanent residence a year after being inspected and paroled, the department said.
Dozens of athletes, many of them from Cuba, have defected during international sports events over the years — some to flee persecution, others to bolster their careers.
Eight members of the Cuban delegation defected when Canada last hosted the Pan Am Games in 1999. Cuban officials at the time alleged that Canadian media were encouraging athletes to jump ship.
When Winnipeg first staged the Games in 1967, a Cuban boxer defected shortly after winning a gold medal.
Two players from the Cuban women's soccer team defected in 2011 after facing off with the Canadian team in Vancouver in an Olympic qualifying match. They crossed into the U.S., where they were reunited with relatives.
And three Cuban soccer players fled to the U.S. the night before a World Cup qualifying match in Toronto in the fall of 2012. They now play for a lower-tier professional league in Charleston, S.C.
Though Cuba's relationship with the U.S. has thawed, some experts have said that at this time, it's unlikely the change will reduce the incentive for Cuban athletes to defect while playing abroad.
Before leaving their team, two of the rowers had won silver medals at these Games. Sotolongo, 18, placed second in the men's quadruple sculls, while Suarez, 25, did in the men's coxless fours.
Turro, Ramos and Suarez also won gold together in the men's lightweight coxless four at the 2011 Pan Am Games in Guadalajara, Mexico.