Holt, a standout player with the Swift Current Broncos from 1989 to 1994, is watching the latest legal proceedings against his former mentor and convicted pedophile closely.
James, who remains in prison serving a sex-crimes sentence, is to appear in a Saskatchewan court this week facing more sex-related charges involving another Broncos player dating back to the early 1990s.
Holt says he encouraged the latest accuser to come forward sooner. But he understands the hesitation.
"It's something you truly think you can take to your grave," Holt recalls. "If I ever had a regret it would be not saying more when I could have."
This is the third time James has faced sex charges.
He served 3 1/2 years in federal prison after he pleaded guilty in 1997 to sex offences against Sheldon Kennedy and two others. Kennedy played for the Swift Current Broncos of the Western Hockey League at the time of the assaults and went on to play in the National Hockey League.
James sought and received a pardon for those offences in 2007, but he was back in court a few years later when Holt and his cousin, retired Calgary Flames star Theo Fleury, came forward with new allegations.
James pleaded guilty and was initially sentenced to two years in 2012. An Appeal Court increased the sentence to five years and, although he has been eligible for full parole, he has never made an application.
Holt says he and scores of other young players were interviewed by police when Kennedy first came forward.
"You don't know what to say. You don't know how to say it. If I did, what were going to be the consequences?
"When you're put on the spot and you've been through this and you've lived the lie for so many years, one more white lie didn't seem like it was that bad. I feel awful that we couldn't be there more for Sheldon."
Both Holt and Kennedy plan to be in Swift Current on Friday when the newest allegations are due back in court.
"It's not about me anymore. It's about these brave warriors — the victims who are coming out," Holt says.
"I had given so much of my life to Graham James. We need to keep our lives going the way we want them, instead of being manipulated or being overshadowed by the darkness of the past.
"I don't think you ever get over it. I think we all continue to heal. I think scars will always remain."
Kennedy, who has dedicated his life to fight sexual abuse, says for those who do speak up the process can be terrifying. He says getting closure isn't going to cure all.
"They think to themselves, 'I've told my story and it's all over now.' But the reality is it's not over. The damage has been done and now the journey starts to get ourselves to a place where we can actually feel good about ourselves," Kennedy says.
"It's clarification that it's not your fault. I think it's one of the biggest burdens that individuals carry — that they've done something to deserve this."
Follow @BillGraveland on Twitter
Also on HuffPost