Dale Romans, trainer of Dullahan, says there are too many horses and too many people in one place at the same time.
"The barn's ridiculous," Romans told a news conference Thursday.
"They could have found such an easier way to accommodate the horse and I don't think anybody who set that barn up and made the rules was thinking about the horse."
Doug O'Neill, trainer of Canadian-owned favourite I'll Have Another, called the barn "an inconvenience for everybody."
All of the 12 horses have been put in a secure barn to ensure there is a level playing field for Saturday's final jewel of the Triple Crown. The barn has security and anything going in is checked.
It looks no different than other barns, other than the security that surrounds it.
Romans said he didn't blame the New York Racing Association, saying it had done the best job it could under the circumstances.
"They fixed the barn up as nice as it could possibly be," he said.
He pointed the finger at the New York State Racing and Wagering Board.
"There's a huge disconnect between our regulators and what's the reality of the (track) backside. That's a major problem in horse racing, in my opinion."
O'Neill said New York regulators are trying to show everyone "how clean the game is and it's not polluted with a bunch of crooks and cheaters like some people have written about."
Romans argued that the detention barn was actually making things worse in the court of public opinion.
"They could have quietly put guards on every horse and never be talked about in the paper. But I think this causes the poor perception and makes people think worse of the game — when this is a very clean sport, it's run properly, there's enough regular checks and balances put into place to make sure it's an even playing field.
"This whole thing isn't really necessary."
Dullahan is the 5-1 second choice in Saturday's race.
"The horses seem to be fine. It's not going to affect the outcome of the race, I don't believe," said Romans. "But it's just not right to take them out of their environment and move them somewhere else three days out from the biggest day of their life."
And Romans suggested the issue of the barn might loom after the race in discussing the result.
"It may affect public opinion after the race. If Doug's horse was to happen to lose, it's going to be human nature for people to say 'Well they got him with the detention barn.' And that's not fair. That diminishes his performance in the first two legs of the Triple Crown and it will also diminish the winner's performance in the Belmont."
All of the Belmont horses had to be in the special barn by noon Wednesday.