Those old memories are being stirred by the current debate over the southern Civil War-era symbol, in the wake of a racist mass-shooting that's prompted a push to banish it from store shelves and public property.
Wilkins was the top-ranking politician in the legislature back then, before he became ambassador to Canada.
Fifteen years ago, he brokered the deal that moved the flag to its current spot. On one side, flag opponents were calling for a boycott of the state. On the other, voters had just turfed a Republican governor who'd proposed removing the flag from its lofty perch atop the state capitol.
Wilkins struck a compromise, urging colleagues to move it to a less-prominent spot nearby on state property. There were tense moments before that arrangement squeaked through the legislature, with a mere six-vote margin.
''There were a lot of threats, made to a lot of people in the state,'' Wilkins said in an interview.
''I got calls. My wife got calls. We didn't react to them, because we never thought we were really in danger. But we did get threats... I got, 'You'll never be Speaker again, and we'll ruin your career.'.. My wife got one one night — it was pretty scary.''
Wilkins went on to Ottawa as George W. Bush's envoy from 2005 to 2009, and the 68-year-old political veteran now works for a Washington-based law firm.
Now the flag issue's back.
The calls for removing the slavery-era symbol are louder than ever after the man accused of killing nine African-Americans in a church was seen in old pictures posing with it.
Walmart removed it from its shelves. Sales then soared on Amazon.com, until the online retailer said it would also stop selling it, as did Sears and eBay.
In South Carolina, Gov. Nikki Haley has promised to recall the legislature this summer unless it takes down the flag; doing so requires a two-thirds vote.
Wilkins said he thinks it'll happen. The politics on the issue have changed since 2000, when there simply wasn't enough support for its full-fledged removal, he said Wednesday in an interview.
''We made a major step 15 years ago. It's time to finish it,'' he said.
''(The flag) still stands for different things for different people. But I think there's a realization now that since it is a hurtful symbol for some, it does not need to be on the statehouse grounds.
''Put it in a museum, and those that want to honour it can honour it.''
Wilkins said as much on his Facebook page, eliciting a broad spectrum of strong opinions — many of them laudatory, others less so.
To him, the emblem symbolizes heritage and the ancestors who died on Civil War battlefields, Wilkins said. But he understands that others look at it and see a painful past.
What's today called the Confederate flag was actually a military battle flag, and was a piece of the national flag of the region that split from the United States in 1861.
There's still plenty of argument about its relationship with slavery — but there is ample historical evidence that slavery was a contributing factor to the split between north and south. The flag was hoisted over the legislature in 1961, for a Civil War centennial that coincided with new north-south tensions over civil rights.
Wilkins said he'd rather have people think of other things when they think of South Carolina.
Like its Indian-American Republican governor, Haley; and Tim Scott, who is the only African-American senator from the south; and the displays of unity after last week's shootings.
''We don't want this flag to define us for people outside the state.''