St. Louis began the series with a 2 1/2-game lead in the NL Central but left with the Pirates ahead by 1 1/2 games. The Cardinals avoided their first sweep in a set of at least five games since 1916, when the New York Giants took all six games.
Pittsburgh had won the previous four games at PNC Park by a combined score of 36-10.
Kelly (2-3) pitched six scoreless innings and had two hits. Tony Cruz added three hits, including a two-run double during an eight-run seventh inning.
Kelly allowed three hits while walking four and striking out four. Seth Maness, Michael Blazek and Edward Mujica pitched one inning each to complete the five-hitter.
After the Cardinals failed to acquire a replacement for injured All-Star catcher Yadier Molina prior to Wednesday's non-waiver trading deadline, Cruz figures to get the majority of starts behind the plate. He had started just 11 games this season until Molina went on the disabled list Wednesday with a sprained right knee.
Carlos Beltran, Allen Craig, Matt Holliday and David Freese also had two of St. Louis' 17 hits. Craig came into the game in a 1-for-27 slump, and he had a two-run double in the seventh.
Charlie Morton (3-3) struggled against the Cardinals again, giving up five runs and 10 hits in six innings. He is 2-8 in his career against St. Louis.
Morton threw a wild pitch that made it 1-0 in the second. St. Louis scored three times in the third as Jon Jay was hit by a pitch with the bases loaded, Cruz grounded into a double play and Pete Kozma had an RBI single.
Beltran doubled in a run in the sixth before the Cardinals broke the game open an inning later.
NOTES: Starling Marte had two of Pittsburgh's five hits. He has a hit in all 12 career games against the Cardinals. ... Pittsburgh C Russell Martin sat out the game with a sore left knee. ... St. Louis opens a three-game series Friday night at Cincinnati with RHP Shelby Miller (10-7, 2.79) facing RHP Bronson Arroyo (9-8, 3.26). ... RHP Gerrit Cole (5-4, 3.56) will pitch for Pittsburgh on Friday night in the first game of the three-game series against Colorado's Jhoulys Chacin (9-5, 3.54).