The week leading into the CFL East Division final has seen members of the Als defence questioned daily on how they'll stop the veteran Toronto Argonauts quarterback now that he is healthy and back on top form.
"I get tired of answering," Brown said Friday. "We've just got to play ball.
"This league is full of good players. We just focus on us and who we are."
The Montreal defence has been taking heat since it started the season by giving up 30 or more points in four straight games, although they won two of those contests.
That they conceded more than 40 points twice after that makes it surprising that they finished 11-7, first in the East Division for the fifth time in seven years.
They will play host to the Argonauts on Sunday at Olympic Stadium with more than 47,000 tickets already sold.
There is a sense the Argonauts have a shot at upsetting Montreal with Ray looking strong down the stretch. Toronto's only win in three regular season meetings with the Alouettes came with Ray under centre, a 23-20 decision in Montreal on July 27.
Ray was injured in the first quarter the next time they met on Sept. 23, a 31-10 Alouettes victory in Montreal. The Alouettes took the rubber match 24-12 on Oct. 14 in Toronto, with Jarious Jackson completing only 20-of-40 passes for the Argos.
"It's exciting to play against Ricky Ray," said defensive end John Bowman. "It's the best against the best, that's how I look at it."
The Montreal defence has been mangled by injuries for a second straight year. In 2011, they lost four of the five defensive backs for most of the campaign but still finished 10-8, although the Alouettes were beaten at home by Hamilton in the division semi-final.
This year, it was the defensive line's turn.
Moton Hopkins hasn't played this season, while fellow defensive tackles J.P. Bekasiak and Ventrel Jenkins went down during the season. Bowman missed a large part of the season. Then replacement Scooter Berry got hurt.
The Alouettes are also missing veteran linebacker Rod Davis, who they had hoped would be back for the playoffs but who has been ruled out for the rest of the post-season.
There is also concern for veteran safety Kyries Hebert, who hurt an ankle in practice on Wednesday. Hebert was back on the field and feeling optimistic Friday, but still needs to be cleared by the medical staff to play.
Defensive lineman Arron Lavarius will come off the practice roster to play his first game as an Alouette on Sunday.
The injuries didn't help a new system and player changes brought in with the hiring of first-year defensive co-ordinator Jeff Reinebold.
"Everyone has been down on our defence all year and we're right here where everyone wants to be — in the East final again," said Bowman. "We don't listen to the naysayers."
The players feel they got better as a the season progressed and point to their final four games, when the team went 3-1. The points they surrendered came mostly after taking comfortable leads.
"We played good football down the stretch, especially in those games where we had already clinched first place," said linebacker Shea Emry. "The eastern final is a different monster.
"You have however many people in here and the place is crazy and everyone's firing on all cylinders. It will be interesting to see what happens."
He looks at the unit's slow start and occasional hapless performance as growing pains.
"That was earlier in the season," said Emry. "We had times when we really didn't know what we were doing out there.
"Now we've come together and everyone's playing together."
But if Montreal's defence was inconsistent, so was Toronto's.
The Alouettes ended up fifth and Toronto sixth in the CFL in points against per game — with 27.2 and 27.3 respectively. The Alouettes were stronger against the run and the Argonauts were better against the pass.
If Montreal's concerned about Ray, the Argonauts will be just as worried by Anthony Calvillo, who led the league in passing yards.
"Anyone who thinks we're going to go in and roll over Toronto is mistaken because they're a tough team," added Bowman. "All we're saying is that it's going to be a good game. We ended on a hot streak too."