Saturday's game in Kingston, Ont., saw one red card and two yellow cards given to the Tongans and one yellow card to Canada.
The Tongan red card was for a punch thrown while the two yellow cards were for shoulder charges that resulted in Matt Evans being stretchered off and James Pritchard requiring medical attention on the field.
Tonga coach Mana Otai said afterwards that Pacific Island teams were being singled out unfairly.
"I had a look at both (yellow cards). I couldn't really see why (they were issued),'' Otai said after reviewing tape of the incidents. "It's almost like, these days you know, when a black man is tackling harder than the other, it seems to be the way.''
Asked about the comments Tuesday, Crowley pointed to three suspensions handed out by the IRB in the wake of the game.
The hit on Evans prompted a six-week ban while the hit on Pritchard earned the offender a three-week suspension, according to Crowley. Another Tongan got a three-week ban resulting from the red card.
The International Rugby Board did not immediately respond to an email on the suspensions.
"I don't know how you could say they were legal tackles," said Crowley. "I thought the definition of a tackle was you had to use your arms for a start. It's pretty clear when you have a look at those incidents that they deserved what they got."
"You hear this story all the time," added the former All Black. "This is about the third or fourth time I've heard it, the racist sort of things. You've got to play within the laws of the game. And if you don't play within the laws of the game, you get dealt with."
While saying he thought the hit on Evans warranted a red card, Crowley said Irish-born referee J.P. Doyle "did a good job" in handling the incidents.
The hangover of the Tongan game is still being felt as Canada prepares to face Ireland at Toronto's BMO Field on Saturday.
The Canadian team is following the IRB concussion protocol on Evans, as a result of the Tongan hit. It's a four-day cycle and Evans failed the first day Sunday.
Sean Duke and Jebb Sinclair, the victim of the punch in the red card incident, are going though the same protocol.
Asked if the punch promoted the concussion, Crowley paused and said: "It's hard to say. I would say it did."
Nick Blevins and Phil Mack (both knee) are also injury concerns.