That's something the government has been working to change — and the effort appears to be paying off.
Tourism arrivals to Haiti rose by 20 per cent in 2013, according to new figures from the country's Tourism Ministry.
Overall, the country welcomed a total of 420,000 tourists.
More than 33,000 of those visitors were from Canada, which is a 60 per cent jump from 2011, when tourism hit at a low a year after Haiti's devastating earthquake.
Haiti's tourism minister, Stephanie Villedrouin, said the government is trying to change, "step by step, the perception of Haiti."
"The goal is to reposition Haiti as a tourism destination and an investment destination," Villedrouin said in an interview.
The Haitian government has made attracting visitors priority, more than doubling the Tourism Ministry's budget to $4.7 million in 2013.
However Haiti has a long way to go.
The country was a popular destination for the jet set in the 1970s, attracting stars like Richard Burton and Elizabeth Taylor.
But years of political instability and, more recently, the devastating 2010 earthquake have changed the country's image and sent most tourists packing.
The country remains well behind the region's tourism leaders such as Dominican Republic, which welcomed 4.7 million visitors last year, and Cuba, which welcomed 2.9 million.
Villedrouin said Haiti isn't trying to directly compete with those countries, which have a well-established set of resorts offering cheap, all-inclusive vacations.
Instead, the country is putting an emphasis on Haiti's culture and history as it tries to draw more visitors — what Villedrouin calls "an all-inclusive country package."
Air Transat, a Montreal-based charter carrier, has been pushing this approach since 2012.
Its trip packages include not only beach time, but also a visit to the Haitian capital, Port-au-Prince, along with guided excursions to historic and natural sites.
The response has been good enough that, last month, it added another offering that includes a visit to the Cap-Haitien, in the north of the country.
"It's clearly not mass tourism," said Renee Boisvert, an Air Transat executive who oversees the company's Haiti trips.
"It's more of a targeted package for people who want more than just sun and the beach."
For now, Haiti doesn't have the infrastructure to accommodate a huge influx of visitors anyway. The country had only 3,200 hotel rooms in 2013, according to the Tourism Ministry.
Many visitors come by cruise ship — 644,000 last year — and stay only for a day, in a fenced-off resort area on the country's north coast.
Haiti's high crime rate complicates things further.
The Canadian government advises travellers to Haiti to "exercise a high degree of caution" due to "high crime rates in various parts of the country and ongoing political tensions."
In an effort to assuage safety concerns, Haiti has introduced a force of so-called "tourism police officers" who speak English and are trained in first aid and customer service.
While some critics have argued Haiti should focus on improving services for its own residents before turning to tourism, Villedrouin said growing the industry provides the country with badly-needed employment and revenue.
Each hotel room built creates two jobs and four indirect jobs, according to Villedrouin. Overall, the government has estimated the tourism push will create more than 1,600 direct jobs and 6,500 indirect jobs.
The government has used tax breaks and other incentives to attract hotel and resort projects. It recently succeeded in bringing international giants such as Marriott and Best Western to the country.
Nancy Roc, a Montreal-based Haitian journalist who has followed the country's tourism industry closely, said the impression of the country among travellers appears to be changing for the better.
"The first thing visitors say is that Haiti does not resemble what they see on international TV," said Roc, who highlights Haiti's must-see destinations at nancyroc.com.
"Port-au-Prince is not Haiti, and that's where the media usually go. It's all the rest of the country with the history and culture that comes with it."