Police said the man didn't appear to have any connections to the trial, which has been underway since mid-April in the Oslo District Court.
Before the incident the man had stopped by the office of Norwegian lawyer Bente Roli, where he dropped off an envelope with "confused" notes, Roli told The Associated Press.
"He was disturbed. Very much. He has nothing to do with the July 22 events," Roli said.
A video clip of the incident posted on Norwegian newspaper VG's website showed the man setting himself on fire before stunned witnesses. He walked up to police with flames shooting up from his sweater and hat, and appeared to yell "shoot me, it hurts" in a Scandinavian language with a foreign accent.
He then ran toward the security checkpoint outside the court but turned right just before the entrance and slumped to the pavement. The man shouted in pain as police officers rushed up to him and tore off his burning sweater.
"There were police officers on site and the fire was put out quickly," police operations leader Finn Belle told The Associated Press.
Roli said she wasn't in the office when the man came in but her secretary met him.
"He was psychologically unstable and thought he was the victim of a conspiracy," she told AP. "He thought everyone was after him because of the social problems he has." She said the man was 49 but didn't name him.
The man was taken to Oslo University Hospital where he was to undergo surgery early Wednesday for serious, but not life-threatening, injuries, police said. They described him as a Norwegian citizen of foreign origin, and with no apparent links to the Breivik trial. "Not as far as we know," police spokeswoman Unni Turid Groendal said.
Nouni Mohsen, a witness who works in a restaurant across the street from the courthouse, told AP that police came in asking for water.
"I didn't have anything but small bottles. They wanted to have a lot of water because there was much fire on his body," Mohsen said.
Breivik has admitted to setting off a car bomb that killed eight people outside the government headquarters in Oslo and then killing 69 people in a shooting rampage at the Labor Party's annual youth camp. The self-styled anti-Muslim militant has shown no remorse for the attacks, saying the victims had betrayed Norway by embracing multiculturalism.
The self-immolation was the first incident outside the court since the trial began. Inside the court, an Iraqi man whose brother was killed in the massacre threw a shoe at Breivik on Friday, hitting one of Breivik's defence lawyers. That incident briefly interrupted the proceedings, which had been conducted without disturbances until that point.
Prosecutors said the proceedings were not interrupted by the drama outside the courthouse Tuesday.
The trial is scheduled to run until the end of June.