The move means the cardinal will not be joining the conclave to choose a new pope, sometime in March.
The cleric issued a statement saying his resignation takes effect immediately. "I do not wish media attention in Rome to be focused on me, but rather on Pope Benedict XVI and on his successor," he said.
Benedict, 85, announced two weeks ago that he was stepping down due to his age and health concerns.
The British newspaper The Observer had reported that O'Brien, 74, who heads the Roman Catholic Church in Scotland, is facing allegations of "inappropriate behaviour."
The paper said three priests and a former priest filed complaints to the Vatican alleging the cardinal approached them in an inappropriate manner.
The Vatican said the resignation of O'Brien as archbishop of Saint Andrews and Edinburgh was accepted under the code of canon law for reasons of O'Brien's age; he turns 75 — the normal retirement age for bishops — on March 17.
O'Brien has said he is contesting allegations.
The one-sentence Vatican statement made no reference to the allegations.