A total of 56 senators voted Thursday in favour of the measure inserted into a bigger highway bill, four short of the 60 needed for approval. Eleven Democrats were among those who wanted the pipeline to move forward as quickly as possible.
Republican support of the proposal was unanimous, but some Democrats were said to have wavered in their opposition after former president Bill Clinton threw his support behind the $7.6 billion pipeline in remarks to an energy conference last week.
His wife, Secretary of State Hillary Clinton, will ultimately decide the fate of the project since it crosses an international border.
President Barack Obama, however, personally contacted some Democratic lawmakers and urged them to vote no. He argued State Department officials had to be allowed the time to do a thorough environmental review of the pipeline.
"He made some calls," White House Press Secretary Jay Carney told reporters before the vote. "The president believes that it is wrong to play politics with a pipeline project whose routes have yet to be proposed ... it cannot possibly be reviewed adequately."
John Boehner, majority leader of the House of Representatives, lambasted Obama for contacting fellow Democrats.
"By personally lobbying against the Keystone pipeline, it means the president of the United States is lobbying for sending North American energy to China and lobbying against American jobs," Boehner told a news conference.
Mitch McConnell, Senate minority leader, called the president "out of touch."
"At a moment when millions are out of work, gas prices are skyrocketing, and the Middle East is in turmoil, we've got a president who's up making phone calls trying to block a pipeline here at home," he said. "It's unbelievable."
Keystone XL would carry millions of barrels of Alberta oilsands crude every week from the northern reaches of the province through six U.S. prairie states to Texas refineries.
Proponents of the pipeline argue it will create jobs and help end American dependence on oil from hostile regimes, while U.S. environmental groups have come together to mount major opposition to the project, calling it a disaster waiting to happen and one that encourages production of what they call Canada's "dirty oil."
Those environmentalists breathed a sigh of relief on Thursday.
"Today's vote was a temporary victory and there's no guarantee that it holds for the long run," Bill McKibben, who led massive White House protests in the summer against the pipeline, said in a statement.
"But given that this thing was a 'no brainer' a year ago, it's pretty remarkable that people power was able to keep working, even in the oil-soaked Senate. We're grateful to the administration for denying the permit and for Senate leadership for holding the line."
Added Jeremy Symons of the National Wildlife Federation: "We are pleased the Senate stood up to oil giants and sided with the American public."
In November, under mounting pressure from environmentalists, the State Department punted making a decision on Keystone until after this year's presidential election, citing concerns about the risks posed by the pipeline's proposed route to a crucial aquifer in Nebraska.
Pipeline proponents cried foul, saying it was a cynical political move aimed at improving Obama's chances of re-election.
Republicans then held the administration's feet to the fire, successfully inserting pipeline provisions into payroll tax cut legislation in late December.
Within a month, facing a mid-February deadline imposed that measure, Obama rejected TransCanada's existing permit outright, saying there wasn't enough time to thoroughly review a new route before giving it the green light.
But Obama also assured Prime Minister Stephen Harper that the decision was not a reflection of the pipeline's merits, but was merely necessitated by Republican pressure tactics. He welcomed TransCanada to propose another route; the company says it will do so soon.
In recent weeks, it appeared the Obama administration's attitude was further softening on Keystone XL after the president praised TransCanada's decision to proceed with constructing the pipeline from Cushing, Okla., to Port Arthur, Tex.
On the campaign trail this week, Republican presidential front-runner Mitt Romney has called the decision a "no-brainer," once again echoing the language once used by Harper to describe the project.
"How in the world can you have a president who doesn't understand the importance of getting energy from our next-door neighbour?" he said.
His campaign, meantime, assailed the president's calls to senators on Thursday.
"President Obama will have plenty of time to lobby Congress on behalf of radical environmentalists when he leaves office next year," read an email from a Romney campaign aide.
"If the president actually thinks that construction of the Keystone XL pipeline is not in the national interest, he should explain his position publicly instead of saying one thing to the American people and another in the backrooms on Capitol Hill."
The chief sponsor of the measure, Sen. John Hoeven of North Dakota, told reporters he's not giving up the fight to force approval of Keystone XL.
"All along we've said the highway bill was just one option. This is a project that got majority support in the Senate. We are making progress," Hoeven said.
"We will see what else comes up, and I'm not even sure that we're done with the highway bill."