It was not immediately clear if the pre-dawn attack on the military base in the southern Abyan province was in retaliation for the death of Fahd al-Quso, a top al-Qaida leader on the FBI's most wanted list.
The militants managed to reach the base both from the sea and by land, gunning down troops and making away with weapons and other military hardware after the blitz attack, Yemeni military officials said.
Government forces later shelled militant positions elsewhere in Abyan, killing 16 militants, said the officials, speaking on condition of anonymity because they were not authorized to speak to the media.
Yemen has been waging an offensive on al-Qaida, whose fighters took advantage of the country's political turmoil during the past year to expand their hold in the south, seizing entire cities and towns and large swathes of land. Abyan's provincial capital of Zinjibar has been held by al-Qaida for a year.
The new Yemeni president, Abed Rabbo Mansour Hadi, has promised improved co-operation with the U.S. to combat the militants. On Saturday, he said the fight against al-Qaida is in its early stages. Hadi took over in February from longtime authoritarian leader Ali Abdullah Saleh.
On Sunday, al-Quso, the top al-Qaida leader, was hit by a missile as he stepped out of his vehicle along with another operative in the southern Shabwa province, Yemeni military officials said.
The drone strike was carried out by the CIA, after an extended surveillance operation by the CIA and U.S. military, two U.S. officials said, speaking on condition of anonymity because they were not authorized to speak to the media.
Al-Quso, 37, was on the FBI's most wanted list, with a $5 million reward for information leading to his capture. He was indicted in the U.S. for his role in the 2000 bombing of the USS Cole in the harbour of Aden, Yemen, in which 17 American sailors were killed and 39 injured.
He served more than five years in a Yemeni prison for his role in the attack and was released in 2007. He briefly escaped prison in 2003 but later turned himself in to serve the rest of his sentence.
A telephone text message claiming to be from al-Qaida's media arm confirmed al-Quso was killed in the strike.
He was also one of the most senior al-Qaida leaders publicly linked to the 2009 Christmas airliner attack and allegedly met in Yemen with the suspected Nigerian bomber, Umar Farouk Abdulmutallab, before the Nigerian left to execute his failed attack over Detroit with a bomb concealed in his underwear.
In December 2010, al-Quso was designated a global terrorist by the State Department, an indication that his role in al-Qaida's Yemen offshoot, al-Qaida in the Arabian Peninsula, had grown more prominent.
Local Yemeni official Abu Bakr bin Farid and the Yemeni Embassy in Washington confirmed al-Quso was killed in Rafd, a remote mountain valley in Shabwa. It is the area where many al-Qaida leaders are believed to have taken cover, including the U.S.-born cleric Anwar al-Awlaki, who was killed in a U.S. airstrike last year.
Al-Quso's association with al-Qaida dates back more than a decade, when he met with Osama bin Laden in Afghanistan. Bin Laden allegedly told him to "eliminate the infidels from the Arabian Peninsula."
From there, al-Quso rose through the ranks. He was assigned in Aden to videotape the bombing of the USS Cole but fell asleep. Despite the lapse, he was declared the regional leader in Aden. He was also believed to have played a prominent role in al-Qaida's attack and the capture last year of Zinjibar.
AP Intelligence Writer Kimberly Dozier in Washington contributed to this report.