Both police and fire departments say they were regularly called to deal with substance abuse problems, medical distress calls and noise complaints at the building.
Deputy Fire Chief Larry Watkinson said there were problems with people in the building and that many of them were not tenants.
But he stopped short of saying whether the troublesome occupants were to blame for the blaze that tore through the two-storey structure.
"It's hard to say, I mean with people coming and going, and not necessarily having a lot of ownership to the building, it doesn't upkeep the best," said Watkinson, who emphasized it was too early to determine the cause of the fire.
Witnesses say the fire at the building in the 33500 block of 3rd Avenue broke out at around 3:30 a.m. PT Friday, destroying at least a dozen of the building's 32 units. There are no reports of injuries.
Debbie Forrest, a former tenant who lived in the building for five years, said she was threatened by other tenants for calling police.
"It definitely catered to the drug addiction people, so that's why I had to move," she told CBC News.
But landlord Steve Yap disputes that claim.
"A new company has taken over for the last five years," he said. "We have really good tenants in there right now."
Watkinson says the building did pass fire inspections, although tenants say there were no sprinklers in the building.