Several boats have been damaged and at least six Vietnamese on board them have been injured, officials said.
Elsewhere in the sea, the Philippines arrested 11 Chinese fishermen for catching endangering turtles, angering Beijing and further exposing regional strains.
China recently has been harassing Vietnam and Philippine vessels and fishermen in the potentially oil- and gas-rich waters it claims almost entirety — a shaky stance to many international law experts.
But China's deployment of the oil rig on May 1 and the flotilla of escort ships, some armed, is seen as one of its most provocative steps in a gradual campaign of asserting its sovereignty in the South China Sea. With neither country showing any sign of stepping down, the standoff raises the possibility of more serious clashes.
Hanoi, which has no hope of competing with China militarily, said it wants a peaceful solution and — unlike China — hadn't sent any navy ships to areas close to the $1 billion deep sea rig near the Paracel Islands. But a top official warned that "all restraint had a limit."
"Our maritime police and fishing protection forces have practiced extreme restraint, we will continue to hold on there," Ngo Ngoc Thu, vice commander of Vietnam's coast guard, told a specially arranged news conference in Hanoi. "But if (the Chinese ships) continue to ram into us, we will respond with similar self-defence."
After China stationed the oil rig, Vietnam immediately dispatched marine police and fishery protection vessels but they were harassed as they approached, Thu said.
Video was shown at the news conference of Chinese ships ramming into Vietnamese vessels and firing high-powered water cannons at them. Thu said the Chinese vessels have done so "dozens" of times over the last three days. He said Vietnam had not carried out any offensive actions of its own close to the rig, around 220 kilometres (140 miles) off the Vietnamese coast.
Chinese Foreign Ministry spokeswoman Hua Chunying told a regular news briefing that the oil rig was in China's territorial waters and therefore drilling is "normal and legal." The country previously said foreign ships would be banned within a 3-mile (4.8-kilometre) radius of the rig.
"The disruptive activities by the Vietnamese side are in violation of China's sovereign rights," she said.
A Vietnamese official who spoke on condition of anonymity because of the issue's sensitivity said earlier that Vietnam's ships were outnumbered by the Chinese flotilla. He said the Vietnamese ships were trying to stop the rig from "establishing a fixed position" at the spot where it wanted to drill.
China's assertiveness, along with its growing military and economic might, is alarming many smaller neighbours even as they are aware they need to keep relations open with a vital trading partner. The United States shares the concerns of the smaller nations.
In Washington, the State Department called China's action provocative and unhelpful to keeping stability in the region.
Asked about the statement, Hua said the U.S. has no right to make unwarranted remarks on China's sovereign rights.
Vietnam has limited leverage in dealing with its giant neighbour. While it is no longer as isolated as it once was, the country can't expect much diplomatic or other help from powerful friends. It appears likely to try to rally regional support against China's actions.
"China seems intent on putting down its footprint squarely in contested waters and force Hanoi's hand. It appears a critical juncture has occurred and one would expect Hanoi to be weighing its options," said Jonathan London, a Vietnam expert at the City University of Hong Kong. "Hanoi's back is against the wall, though China's policies — which according to virtually everyone except China are baseless legally — have brought about this situation."
The Philippines has filed a legal challenge to China's territorial claims at a U.N. tribunal, against the wishes of China. Vietnam and other claimant states haven't done that yet.
Tran Duy Hai, vice chairman of Vietnam's national borders committee, didn't rule it out.
"Vietnam will have to use all measures stipulated in the U.N. Charter to defend its interests," he said.
The arrests of the 11 fisherman Wednesday by the Philippines took place near territory known as Half Moon Shoal in waters claimed by Beijing and Manila. China demanded that the Philippines release the boat, and Hua Chunying, the foreign ministry spokeswoman, urged Manila to "stop taking further provocative actions."
Philippine maritime police Chief Superintendent Noel Vargas said the fishermen will face charges of violating Philippine laws prohibiting catches of endangered green sea turtles.
China occupied the Paracel Islands 40 years ago, and 74 U.S.-backed South Vietnamese forces died in a subsequent military clash. The Vietnamese and Chinese navies clashed again in 1988 in the disputed Spratly Islands, leaving 64 Vietnamese sailors dead.
In 1992, China awarded a contract to U.S. energy company Crestone to explore for oil and gas in the Spratly Islands. Vietnam protested the move. Two years later, Vietnam's navy forced the company's oil rigs to leave.
Associated Press writers Louise Watt in Beijing and Jim Gomez in Manila contributed to this report.