English referee Martin Atkinson interrupted the match in the 41st minute when Serbia defender Stefan Mitrovic managed to grab the banner and Albania players tried to protect it. Several Serbian fans invaded the pitch and clashed with Albania players.
The Group I match was later abandoned "due to a disturbance on the pitch," according to UEFA, with the score at 0-0.
"The Euro 2016 Qualifying match between Serbia and Albania has been abandoned and the circumstances will be reported to the UEFA Disciplinary Body," European football's ruling body said in a statement.
Albanian fans had been warned against attending the game between the two Balkan rival nations that have had turbulent relations — mainly over the former ethnic Albanian-dominated Serbian province of Kosovo that declared independence in 2008.
"We wanted to continue the match, but Albanian players said they were not psychologically ready," Serbia captain Branislav Ivanovic said.
Both Serbian and Albanian fans have a long history of violence and racial abuse. In October 2010, the Italy-Serbia European Championship qualifier was disrupted in Genoa by violent Serbia fans. UEFA eventually awarded Italy a 3-0 win.
On Tuesday, the Albanian anthem was loudly jeered by Serbian fans and derogatory chanting was heard throughout the first half. Serbian supporters also threw flares at the pitch.
Albania captain Lorik Cana said he and his teammates were hit by objects thrown from the stands and Albania's Italian coach Giovanni de Biazzi said four players were injured.
"We also complained (to the UEFA delegate) that security staff in the stadium attacked us physically, as well as fans and some players," he told Albania's Supersport private TV channel.
The drone, clearly visible in the lights of the stadium, made a series of passes above the pitch.
The banner displayed an Albanian flag and a map of so-called "Greater Albania," an area that comprises territory within today's Albania, Kosovo, Serbia, Montenegro, Macedonia and northern Greece.
The banner also portrayed two Albanian nationalist leaders — Ismail Qemali, who declared Albania's independence from the Ottoman Empire in 1912 and Isa Boletini, an Albanian fighter against the Turks.
Fearing trouble, hundreds of riot police were deployed inside and outside the Partizan stadium in Belgrade. Dozens of Albanian fans, mostly from Kosovo and Montenegro, apparently managed to get inside despite warnings by Serbian security officials that they would be arrested if they carry Albanian symbols.
On his Twitter page, Albanian Prime Minister Edi Rama congratulated Albanian players for their courage before the match was abandoned and added that he was "sorry for the neighbours who left a bad worldwide image with the ugly show."
Thousands of Albanian fans who followed the match on a big screen in Tirana headed for the airport in the Albanian capital to welcome the players on their arrival from Belgrade.
After the match, Albanian Interior Minister Saimir Tahiri sent a letter to his Serbian counterpart expressing concerns for the security of the Albania players, team staff, journalists and a few fans.
The minister asked for "urgent steps to guarantee the security of all Albanian citizens present in the stadium."
Associated Press writer Llazar Semini contributed from Tirana, Albania.