President Enrique Pena Nieto's Institutional Revolutionary Party, or PRI, and two allied parties had about 40 per cent of the vote, according to preliminary results with 78 per cent of the ballots counted.
But that may be enough to preserve their narrow majority in Congress, partly due to divisions among rival parties.
"The PRI lost, but not very much," said Jesus Cantu, political analyst at the Monterrey Institute of Technology.
In an election marred by sporadic violence, independent Jaime Rodriguez, known as "El Bronco," won the governor's race in the border state of Nuevo Leon, ousting the PRI from a key state that includes the business hub of Monterrey. His popularity was attributed to voters' disgust with all political parties.
"I think in the whole country, this will help the political parties to renew and transform themselves so they can be better," said Rodriguez, adding that he would give them a "six-year vacation," a reference to the length of his term in office.
Rodriguez said his first action as governor would be to attack corruption: "We have to investigate the entire previous government."
It was the first election in Mexico to allow unaffiliated candidates, thanks to electoral reform last year.
The horseback-riding, boot-clad, tough-talking Rodriguez earned his nickname after he survived two assassination attempts that left his car bullet-ridden as mayor of a suburb of Monterrey. He said the attacks were from a drug cartel.
His support harkens back to 2000, when another plainspoken cowboy candidate, Vicente Fox, managed to topple the PRI's 71-year rule and win the presidency for the opposition National Action Party.
Sunday's vote came amid widespread discontent with politicians in Mexico, where corruption scandals, a lacklustre economy and human rights concerns related to missing students and suspected army massacres have tarnished Pena Nieto's image and fed anti-government protests.
Pena Nieto was boosted by a bump in support for the allied Green Party, which jumped from about 6 per cent of overall support to 7 per cent following a controversial campaign in which it was fined millions for ignoring campaign finance laws. That could give the Greens as many as 20 new seats in the 500-member lower house of Congress.
The biggest loser was the Democratic Revolution Party, which has been Mexico's leading leftist party for a quarter century. It won just over 10 per cent of the vote because the candidate who nearly led it to the presidency twice, Andres Manuel Lopez Obrador, bolted and formed a new movement, pulling away millions of followers. His Morena party won about 9 per cent. The split apparently cost the left several seats.
Thousands of soldiers and federal police guarded polling stations where violence and calls for boycotts threatened to mar the vote for 500 seats in the lower house of Congress, nine of 31 governorships and hundreds of mayors and local officials.
Protesters burned ballot boxes in several restive states in southern Mexico, but officials called the disruptions "isolated incidents." A statement from a team of election observers from the Organization of America States, headed by former Costa Rican President Laura Chinchilla, said the incidents didn't prevent people from voting.
"There were those who wanted to affect the elections, including with violence in the previous days designed to discourage the public," Pena Nieto said in a national address. "But the mandate Mexicans gave to authorities today was to reject violence and intolerance."
A loose coalition of radical teachers' unions and activists had vowed to block the vote, and protesters burned at least seven ballot boxes and election materials in Tixtla, the Guerrero state town where 43 students vanished at the hands of a local police force, creating national outrage.
Ballot boxes were also destroyed in the southern states of Chiapas and Oaxaca. In Oaxaca's capital, masked protesters emptied a vehicle of ballots, boxes and voting tables and burned the material in the main square.
The state government reported 88 arrests related to the destruction of election materials and disturbances in the capital, Tuxtepec and Salina Cruz.
In Monterrey, two political parties reported that armed men were intimidating voters in three towns near the border with Texas.
Violence ahead of the elections claimed the lives of three candidates, one would-be candidate and at least a dozen campaign workers or activists.
Associated Press journalists Jose Maria Alvarez and Jose Antonio Rivera in Tixtla, Mark Stevenson in Oaxaca, Porfirio Ibarra in Monterrey and Christopher Sherman, Maria Verza and Peter Orsi in Mexico City contributed to this report.
This story has been corrected to show the last name of the former Costa Rican president is Chinchilla, not Chincilla.Suggest a correction