Moments after Everton keeper Joel Robles had hit the underside of the crossbar at Upton Park, Adrian stepped up to convert his spotkick and end a match that had been deadlocked at 1-1 after 90 minutes and 2-2 following extra time.
Enner Valencia put West Ham in front with an angled shot in the 51st minute and Everton's night took another turn for the worse five minutes later when Aiden McGeady was dismissed for a second yellow card. But Kevin Mirallas levelled in the 82nd minute, curling a superb free kick around the West Ham wall to send the match to extra time. Mirallas then turned provider with a mazy run and a clever pass that gave striker Romelu Lukaku an easy finish from close range in the 97th.
Substitute Carlton Cole equalized for the hosts in the 113th, forcing the shootout.
"It was an unbelievable match, wasn't it? You can't say the FA Cup has died after that, not with that entertainment value," West Ham coach Sam Allardyce told broadcaster BT Sport.
"I thought today had absolutely everything a cup tie should have — a little bit of controversy, some great play, chances missed, some great goalkeeping and some very good goals."
West Ham's reward is an away fixture against third-tier Bristol City, which beat Doncaster 2-0 in another of Tuesday's replays.
Fulham also needed penalties to reach the fourth round, winning its shootout 5-3 against Wolverhampton Wanderers after the match had finished 3-3 following extra time.
Second-tier Fulham will now face a Premier League opponent, Sunderland.
Chesterfield beat Scunthorpe 2-0 in the night's other replay.