Ki Sung-yueng scored the winning penalty after Daniel Sturridge missed the fifth penalty for Britain, which hadn't fielded a men's Olympic soccer team since 1960.
The score was 1-1 after regulation time, and neither team scored in the 30-minute overtime.
South Korea will play its first Olympics semifinal in soccer against Brazil, which won its quarterfinal earlier Saturday, 3-2 against Honduras.
South Korea's Ji Dong-won scored on a long-range shot at the 29-minute mark, but Aaron Ramsey tied the score on a penalty six minutes later. He could have given Britain a 2-1 lead not long after, when Britain was given another penalty, but his shot was saved.
Sturridge, who plays for Chelsea in the English Premier League and was one of the sharpest British players of the night, paused for a second in his run-up to his left-footed penalty shot, then kicked the ball toward the right side of the goal. Lee Bum-young, a substitute goalkeeper, guessed correctly, diving to punch the ball away.
That left it up to Ki to give South Korea the victory. The forward, who plays for Celtic of the Scottish Premier League, shot powerfully into the top left-hand corner of the net, and Britain's hopes for an Olympic medal in the sport it invented were over.
There's no guarantee that the nation will field another unified team. With opposition from the football associations of Wales, Scotland and Northern Ireland, few see any chance of another team at the 2016 Rio Games.
South Korean coach Hong Myung-bo said he had a "strong belief" his team would prevail because he knew English teams had problems with penalty shootouts.
"The only thing that I told them was to keep calm and kick the ball with confidence," he said. "I asked the players to make it very difficult for (Britain) to play its own game."
For the English players on Britain's team, going out of a major competition on penalties shouldn't be a surprise. England's penalty record stands at six defeats and one victory from seven shoot-outs at major tournaments. British coach Stuart Pearce himself missed against Germany in the 1990 World Cup semifinal.
"The first day we got together as a group we practiced penalties," Pearce said. "Daniel (Sturridge) got the winner three days ago that got us out of the group stages, and today he misses the penalty. You have days like that, but he will benefit from it, if my experience is anything to go by."
Manchester United star Ryan Giggs, who scored in the shootout, pointed out that the team had played only about a half-dozen games together.
"The Korean lads had, I think, played 18 games unbeaten, so that's the sort of preparation they've had compared to our preparation," Giggs said. "We've got better each game, but it was a bit difficult for us today."
After Ji's goal, Britain was trailing for the first time in the competition but tied the score soon after.
Craig Bellamy found space on the right and pulled the ball back to Ryan Bertand, whose shot hit the hand of Oh Jae-suk. Overruling noisy South Korean protests, referee Wilmar Roldan awarded the penalty. Aaron Ramsey hit a low shot to the left that squeezed under the keeper.
Within minutes, Britain had the chance to take the lead when referee Roldan again pointed to the penalty spot for a foul on Sturridge by Koo Ja-cheol. This time Ramsey wasn't so lucky. He again kept the ball low, hitting it to the right, but the shot was easily saved.