Now the U.S. Hall of Famer is saddling the top two favourites for the Kentucky Derby. Heady stuff for the kid who grew up watching the race on television in his border hometown of Nogales, Arizona.
"Destiny," he jokingly proclaimed on Thursday outside his cement-block barn at Churchill Downs.
Baffert has early 5-2 favourite American Pharoah and 3-1 second choice Dortmund in what some believe is the strongest 1-2 punch since trainer Ben Jones won with Citation and finished second with Coaltown in 1948.
"I'd love to see them 1-2 turning for home and then see what they're made of," Baffert said.
He experienced that scenario in 1998, when Real Quiet and Indian Charlie were 1-2 headed for home. Real Quiet won, but Victory Gallop broke up a Baffert exacta; Indian Charlie finished third.
It was the second of Baffert's three Derby victories. He had two firsts and a second in the first three years he entered 19 years ago; his last victory was with War Emblem in 2002.
"I thought, 'This is fun, this is easy,'" the white-haired, 62-year-old trainer said. "When you get older, you start appreciating it more, and you realize, 'Boy, this is really tough.'"
Quite a difference from his humble beginnings, when he started with one horse. His brothers and a friend cheered him on at his first few races in Southern California. Baffert's horse was badly beaten and they declined his invitation to come back again.
Baffert didn't apprentice under an established thoroughbred trainer, the usual preparation before going out on your own.
"I had to learn by trial and error, which most of it was error," he said. "It was really good for me, though, because it really made me work much harder and it's kept me hungry all these years."
Baffert is as eager as everyone else to see which is better: American Pharoah or Dortmund. American Pharoah brings a four-race winning streak into the 141st Derby; Dortmund is 6-0
"Until they get with all the other horses, you just don't know what you have," Baffert said.
American Pharoah has glided over the Churchill Downs strip since arriving direct from an eight-length win in the Arkansas Derby.
Dortmund has taken a better liking to Churchill's dirt surface than his home track at Santa Anita in California. He's unusually tall (standing 17 hands or nearly 5-foot-8) and weighs 1,280 pounds (580 kilograms).
"He's light on his feet for a big horse," Baffert said.
American Pharoah, by comparison, is just over 16 hands and weighs 1,170 pounds (530 kilos).
Both colts go into the 1 1/4-mile (2,000-meter) Derby with big targets on their backs.
"Anytime Bob Baffert has a couple of Kentucky Derby horses, you have to give them a lot of respect," said rival trainer Todd Pletcher, who will saddle three horses, including 8-1 third choice Carpe Diem. "He's certainly tremendous at getting them there and having them perform well."
Yet Baffert knows the sting of losing, too.
He thought he had a couple of aces in 2001, with 9-5 favourite Point Given and third choice Congaree. But Point Given, who went on to win U.S. Horse of the Year, finished fifth and Congaree third.
"The owners, they can dream in Technicolor," he said. "I dream in black and white. I don't dare get too ahead of myself."