In a throne speech read by Lt.-Gov. J.J. Grant, the government says the panel is a recognition of allegations made by former residents of the Nova Scotia Home for Colored Children.
"Allegations of abuse must be taken seriously, with due concern for victims and for justice," the speech says.
"My government agrees that action must be taken. In the coming weeks, terms of reference for an independent panel will be developed in consultation with members of the African Nova Scotian community."
Tony Smith is among former residents who have pushed the province for a public inquiry and he was disappointed by the government's decision to appoint a panel to hear the allegations, which date back 50 years or more.
"It does nothing to address the healing process," he said, adding he wouldn't participate in the government's panel based on Tuesday's announcement.
"Are they going to hold my hand and say, 'There, there, everything's better?' I have no idea."
Former resident Tracey Dorrington-Skinner said the panel isn't "near enough" for former residents.
"Not only is it unfair, but it's not justice at all," she said.
The announcement of the panel was the most significant piece of the speech, which largely listed the NDP government's accomplishments since it came to power almost four years ago and with an election expected this year.
The speech also promised the creation of a Department of Education and Early Childhood Development, which the government said would better co-ordinate services for infants, young children and their families. The new department would draw on staff from the departments of Education, Community Services, and Health and Wellness.
It said the government would continue to expand the role and the number of nurse practitioners, and introduce legislation that stiffens penalties in animal abuse cases.
Before the throne speech, Dexter said the government's commitment for the independent review of the orphanage would allow people who allege they were abused to speak publicly about their experiences.
The opposition leaders criticized the plan because it falls short of an inquiry, but Dexter said the panel was the best way of addressing what members of the black community wanted.
"They wanted one of reconciliation and one of healing," said Dexter. "It is really about a community wanting to have some recognition of the things that happened in the past."
Liberal Leader Stephen McNeil said the panel will lack the weight of a judicial inquiry.
"The victims of this potential crime were looking for help and support when they were children and none of us listened. Quite frankly, today, I don't believe this government has listened."
Progressive Conservative Leader Jamie Baillie said the issue of a public inquiry is one of "fairness and justice."
"I'm disappointed that he (Dexter) is offering them less than they deserve," said Baillie.
In December, Halifax police and the RCMP announced they wouldn't be laying criminal charges in the case after concluding there wasn't enough evidence.
A class action lawsuit involving 140 former residents, who say they were physically, sexually and mentally abused by staff at the home over several decades until the 1980s, is scheduled to resume this week in a Halifax court.