The Libyan government said on the prime minister's website that he was taken "to an unknown destination for unknown reasons."
A further statement on the government's Facebook page said that the country's cabinet had no knowledge of any arrest warrants for him.
Guards at the Corinthia Hotel said the gunmen took Zeidan from the hotel but there were no shots fired or clashes during the incident.
One guard described it as an "arrest" while another told Reuters the men were militants.
Al-Arabiya television channel quoted Libya's justice minister as saying that Zeidan had been "kidnapped" and showed what it said were video stills of Zeidan frowning and wearing a grey shirt undone at the collar surrounded by several men in civilian clothes pressing closely around him.
Still another report, from the Bloomberg news agency, cited a top Tripoli security official saying the prime minister had been mistakenly arrested by a group called the Libya Revolutionaries Operations Room, which believed there was a warrant for him. The group is a former rebel faction that fought to overthrow former Libyan leader Moammar Gadhafi and now does work for the country's Interior Ministry, Bloomberg reported.
Government officials could not immediately be reached to give more details.
Two years after a revolution toppled Gadhafi, the fragile central government has been struggling to contain rival militias and Islamist militants who control parts of the country.
Last weekend, the U.S. military captured a top al-Qaeda suspect in Tripoli in a raid that angered Islamist militant groups, including one blamed for the 2012 attack on the U.S. consulate in Benghazi.