But it wasn't until the Southern Foodways Alliance contacted him to make a documentary short about his restaurant that Lee fully realized the Big Apple Inn's place in the history of Southern cuisine.
"I'm keeping the legacy of those generations alive," Lee said.
"Smoke & Ears," a documentary about the Big Apple Inn by filmmaker Joe York, is one of the alliance's many oral history projects. The Oxford, Miss., non-profit is part of the University of Mississippi's Center for the Study of Southern Culture. The SFA seeks to preserve the rich history of Southern food while starting conversations about the cultural forces behind staples like barbecue, collard greens and sweet tea.
"In many ways, food is a nonthreatening entry point to speaking about bigger issues like race, class, gender and identity," said John T. Edge, a food writer and director of the SFA.
In the 1920's, Lee's great-grandfather, Juan Mora, came to the United States from Mexico. During the 1930's he started selling tamales from a tin drum over an open fire in the Farish Street area, then the centre of African-American life in segregated Jackson.
Mora soon opened the Big Apple Inn, named after a popular dance, and expanded the menu.
Today, the working-class restaurant serves Mississippi-style tamales — made with ground beef or turkey instead of shredded pork or beans — smoked sausage sandwiches with coleslaw and hot sauce, pig ear sandwiches, hot dogs, hamburgers and bologna sandwiches.
The Big Apple is one of just a few businesses in an otherwise blighted area, where faded signs on vacant storefronts hint at a once-vibrant scene. In decades past, the restaurant was one of the area's many minority-owned businesses. Civil rights leader Medgar Evers had an office above the Big Apple Inn and met with activists inside.
Lee's mother, Mary Harrison Lee, was one of the Tougaloo Four, a group of civil rights era Freedom Riders arrested when they bought bus tickets to New Orleans and sat down in a whites-only waiting room to protest segregation. His father, Gene Lee, was a fishing buddy of blues singer and harmonica player Sonny Boy Williams, who lived in an apartment above the restaurant.
The multi-racial family's role in civic affairs and their cross into culinary history is not as improbable as it seems, Edge said. That's because Southern food has long blended African and European traditions, along with elements of Asian and Latin American flavours.
Edge said such projects pay respect to the South's nuanced food heritage, without pretending it doesn't have problems.
"We're not interested in preserving the past in amber," Edge said. "To my mind the South is changing for the better. There's an idea that traditions are the innovations that succeed. I like that."
The SFA also works with the University of Mississippi to bring scholars of food traditions, like history professor Jill Cooley, to the state. Cooley teaches classes that incorporate lessons on food history, economics and culture at the University of Mississippi.
Cooley says that reality of what common people cooked and served in their kitchens sometimes clashes with romantic visions of Southern cuisine.
"The image of the bountiful table is a myth," Cooley said.
Poor whites and African-Americans were often malnourished and wound up with lesser cuts of meat during the days of slavery and sharecropping, leaving people to make do with chitlins and pig feet, Cooley said.
"The economic system left people out of being able to access good, healthy food in plentiful supply. Often times there were people manipulating the food supply to maintain a lower class of workers," she said.
Obesity is also a serious concern throughout the South, and while many blame the tradition of fattening fried foods and sugary tea, Edge said good Southern cooking tends to be healthier than the fast-food version of today. A typical Southern lunch has long been the "meat and three" — pulled pork, fried chicken or catfish, for instance, along with three vegetable sides.
The Big Apple Inn has never claimed to serve healthy food, and many of its offerings were partly born out of frugality. When Mora opened the restaurant, he was offered a box of the pig ears and figured out a way to cook them so they were palatable. Today, they've become a local delicacy.
In "Smoke & Ears," Big Apple customers recall receiving sandwiches instead of candy on Halloween. They sing songs about smoked sausage on a bun, homemade hot sauce and slippery pig ears that have been boiled into submission.
Lee knew the Big Apple was a popular place, but he didn't anticipate the interest for the documentary or the award he received from the SFA for keeping a Southern food tradition alive.
Since then, Lee has played the documentary to customers as they waited for their orders.
"Is this about a pig ear sandwich?" Edge said. "No, there's a lot more going on here than a sloppy piece of pig ear hanging from a bun."