Norwich, which is 85 places above fifth-tier club Luton in the English football pyramid, was beaten 1-0 at home after Scott Rendell scored in the 80th minute.
"It's an unbelievable feeling to come to a Premiership club like Norwich and turn them over," Rendell said. "It's fantastic to be part of history. We were aware of it."
The FA Cup has not witnessed such a giant killing since Sutton beat Coventry 2-1 in the third round in 1989.
And not since 1949 has a non-league team managed to overcome a side from the top flight beyond the third round.
Luton, FA Cup semifinalists in 1994, won the League Cup in 1988 and was in the top flight until 1992 before dropping down the divisions and enduring financial turmoil.
Twenty years ago, Luton goalkeeper Mark Tyler was released by hometown club Norwich as a schoolboy.
"The hard part about today was the emotional side, coming back here playing," the 35-year-old Tyler said. "It's where I started. This is where I wanted to be as a little boy playing football.
"To come back here, play, keep a clean sheet, means the absolute world to me."
Norwich chief executive David McNally said on Twitter: "Like all fans, I am embarrassed and hurt."
Norwich manager Chris Hughton defended his decision to make six changes to the team, which is without a win in six Premier League matches
"We pushed and pushed and they caught us on a classic counterattack," Hughton said. "It was a poor goal for us to concede."
The day's other upset saw MK Dons win 4-2 at Queens Park Rangers, which is bottom of the Premier League.
"Getting into the fourth round was a little bit of history for us and now the fifth round makes it a little more special," Dons manager Karl Robinson said.
The Dons were gifted the lead when Armand Traore scored an own goal, Ryan Lowe extended the lead before the break, while Ryan Harley and Darren Potter scored at the start of the second half.
QPR manager Harry Redknapp had made nine changes to the team, which scored late consolations through Jay Bothroyd and Fabio da Silva.
"I gave them a chance today because they knock on my door and say they should be playing and other people tell me they are all good players," Redknapp said. "Well, they got their chance today and they blew it ... That is why I have gone all over Europe these last three weeks trying to improve the squad because we are short."
Manchester United had no trouble overcoming Fulham, with the Premier League leaders winning 4-1 to advance to the last 16, while Arsenal was twice pegged back at Brighton after taking the lead but won 3-2 at the second-tier club thanks to a late goal from substitute Theo Walcott.
Olivier Giroud put Arsenal ahead in the 16th minute but Ashley Barnes levelled with a powerful header.
Giroud restored Arsenal's lead at the start of the second half, and Leonardo Ulloa drew Brighton level again with a diving header, but Walcott had the final say in the 85th.
Ryan Giggs put Man United ahead from the penalty spot in the 3rd after Aaron Hughes handled.
Wayne Rooney doubled United's lead with a fierce strike at the start of the second half before Javier Hernandez scored twice.
But United has only registered seven clean sheets this season, and Hughes was able to head in a late goal for Fulham at Old Trafford.
Manchester City scored with five minutes to go at Stoke to advance with a 1-0 victory. Pablo Zabaleta neatly stabbed the ball into the net in the 85th minute after Edin Dzeko clipped the ball across from Sergio Aguero's pass.
But there was a setback for City with Vincent Kompany limping off with a calf injury in the first half, and the captain will miss the Premier League match against QPR on Tuesday.
Reading advanced easily with a 4-0 win over Sheffield United, and Wigan avoided embarrassment by beating non-league side Macclesfield 1-0 through Jordi Gomez's penalty kick.
Everton's 500th match in David Moyes' managerial reign ended in dramatic fashion with substitute Johnny Heitinga scoring in stoppage time to clinch a 2-1 win over Bolton.
Steven Pienaar's opener for Everton had been cancelled out by Marvin Sordell.