Mark Twain once said that there are three types of untruths: Lies, damned lies, and statistics.
Friday’s job report might be some pretty powerful evidence of the truth of that adage.
Canada’s job numbers for May surprised all the experts. The consensus call was for some 10,000 new jobs, but the jobs report beat that by a mile: There were 95,000 jobs created in Canada in May, almost all in full-time, private-sector positions.
That’s great news for Canada — if the numbers are right. And the odds are, they aren’t spot on.
“Canada’s jobs market isn’t booming the way the [labour force survey] is suggesting and the stellar jobs report should be interpreted with caution,” National Bank Financial said in a report Friday morning.
BMO chief economist Douglas Porter had a similar take.
“The volatility in Canada’s employment reports puts a massive warning label on this release,” he wrote. “Note that even with the big move, the three-month trend is actually a bit below average for job growth. Shiny, sparkly headline, dull, dreary trend.”
Many economists have complained that the monthly labour force survey is a rough and often unreliable gauge of the job market. The first problem with it is that it’s a survey, not much different opinion polls and customer surveys, which is used to estimate the strength of the entire workforce
In the case of the labour force survey, StatsCan calls 55,000 households to poll them on their job situation. While that sounds like a large sample, it actually yields a very high margin of error.
StatsCan pegs that margin of error at 53,400 jobs, 19 times out of 20. So that means the number of jobs created in Canada in May was anywhere between 41,600 and 148,400---still a major gain and much bigger than economists projected. Still, there is a one-in-20 chance the real number isn’t even in that range.
In months when job growth is lower, the data can seem even more distorted. For instance, in April, StatsCan said the economy created 12,500 jobs. But given the margin of error, the country could have actually lost 40,900 jobs, and the number would still be considered statistically valid.
Not exactly a reliable indicator.
Economist Will Dunning points out that this margin of error is actually larger than the typical change in job numbers from month to month, which he pegs at around 15,000 or 20,000. When the margin of error is larger than the numbers being measured, the data isn’t of much use.
“We should not believe the data from month to month,” he concluded in an email to HuffPost.
Dunning, an economic consultant who also acts as the chief economist for Canada's mortgage broker lobby, also says the unemployment numbers for Canada’s largest cities are particularly unreliable because residents of those cities are underrepresented in the survey.
“Toronto has about three per cent of the national sample but about 16 to 17 per cent of the population. This means that there are large ‘standard errors’ [in the way the sample was calculated] in Toronto, Vancouver, and Montreal,” he wrote.
Looking at the breakdown by city, Dunning suggests this month’s numbers were distorted “by unusually large estimates for Montreal and Toronto.”
Indeed, Montreal appears to have create a whopping 21,900 jobs in May, or one-quarter of the national total. And Toronto accounted for another 26,000 jobs. Between the two cities, that's half of Canada's job growth in May. Is this even possible?
Dunning highlighted a simple way out of the problem: Stop paying attention to the labour force survey, and start paying attention to the more reliable Survey of Employment, Payrolls and Hours, which still relies on a survey, this time of businesses, but also relies on actual payroll data from Canada Revenue Agency, and therefore “provides less volatile and more believable data.”
However, it lags behind the labour force survey numbers by two months, so by the time it comes out people have moved on to more recent (and less reliable) jobs numbers.
So thanks to our collective impatience, we'll probably keep paying attention to a wholly unreliable measure of our economic health.