The study was aimed at preventing repeat heart attacks and strokes in people who had already suffered one or were in danger of one because of hardened arteries in their legs.
The drug, vorapaxar, lowered the risk of those problems but also raised the risk of major bleeding, including dangerous bleeding in the head, which largely cancelled out the drug's benefit.
Results of the study were discussed Saturday at an American College of Cardiology conference in Chicago and published by the New England Journal of Medicine.
Merck had hoped vorapaxar would become a new, first-of-its-kind blood thinner.
The company-sponsored study involved more than 26,000 patients in 32 countries. All were given usual heart medicines plus aspirin, and half also received daily vorapaxar pills.
Safety monitors stopped part of the study last year after seeing higher rates of bleeding in the head among people with a history of stroke who were on the experimental drug. The study continued in the rest of the participants.
After three years, about 9 per cent of those given vorapaxar had suffered a heart attack or a stroke or had died from heart-related causes versus more than 10 per cent of those not given the drug. Moderate or severe bleeding occurred in about 4 per cent of those on vorapaxar versus just more than 2 per cent of the others, said the study's leader, Dr. David Morrow of Brigham and Women's Hospital in Boston.
Among those with a history of heart attack — two-thirds of study participants — the drug had a net benefit, though, leaving the possibility that Merck might pursue seeking federal approval to sell it for these patients.
However, several experts not connected with the study said the drug's relatively modest benefit and likely high cost would make it a tough sell even if it were allowed on the market.
"This is not a drug that I would put in my personal medicine chest," said Dr. Eduardo Marban of Cedars-Sinai Medical Center in Los Angeles.
Merck officials said they would discuss the results with more scientists before deciding next steps.
Heart meeting: http://www.cardiosource.org
New England Journal: http://www.nejm.org
Marilynn Marchione can be followed at http://twitter.com/MMarchioneAP