Speaking at Anahuac University in Mexico City, Pele said he sides with those protesting against the big spending, but also laments any violence.
Brazil is spending about $11 billion overall on the World Cup, and $4 billion alone on 12 new and refurbished stadiums.
"It's clear that politically speaking, the money spent to build the stadiums was a lot, and in some cases was more than it should have been," Pele said.
He said "some of this money could have been invested in schools, in hospitals.... Brazil needs it. That's clear. On that point, I agree (with the protests). But I lament what protesters are doing, which is breaking and burning everything. It's money that we will have to spend again."
The 73-year-old, who helped Brazil win three of its five World Cups, criticized the late preparations of stadiums, and said the confusion was likely to keep tourists away.
"We already know that 25 per cent of foreigners who are going to Brazil (for the World Cup) are worried about this movement (protests), and I think they have even cancelled (their trip). So this is a great loss for the country," he said on Monday.
Pele was particularly critical of the stadium in Sao Paulo, the venue for the opening match between Brazil and Croatia on June 12. Officials acknowledged last week that a portion of the roof won't be completed until after the World Cup ends on July 13.
"It's a pity that we still have a stadium that's not yet finished,' he said. "The first (World Cup) match is going to be in the Corinthians stadium in Sao Paulo, which is called Itaquerao, but there's a problem because it's not finished yet. That's what I say is regrettable. That's a problem."
Several other stadiums are also still being finished with the World Cup just over three weeks away.