The initial miscalculation of Statistics Canada’s jobs numbers for July was related to a major redesign of Canada’s Labour Force Survey (LFS), as well as employees’ “incomplete understanding” of processing systems, according to an investigation into the error by the agency.
The review issued Thursday stemmed from Statscan’s correction of the most recent LFS, which misreported the country’s hiring growth. The updated figure showed a gain of nearly 42,000 jobs in the month, from an initial report of just 200 jobs.
Statscan revealed this initial low result was challenged and discussed internally, but that the figures were deemed “plausible in the current economic context and the sampling variability inherent to the LFS.” When staff looked back at surveys over about 30 years, they found that the LFS showed a low net increase of 200 jobs or less nearly 20 per cent of the time when total employment is growing.
A temporary half-cent drop in the Canadian dollar followed the initial jobs report on Aug. 8, and caused Ottawa to hold up Employment Insurance claims, since benefits are tied in part to regional unemployment rates. When Statscan corrected its mistake one week later, currency rebounded to 92 cents U.S., from 91.7 cents before the update was released.
“I find it quite telling, and I think it should inform everyone here about the nature of the survey, that a decline of almost 60,000 full-time jobs as the initial report said was seen within the normal range,” said Douglas Porter, chief economist at BMO Nesbitt Burns.
Statscan provided a deeply detailed account of how the error came to be.
The LFS is in the midst of a redesign that occurs once a decade. In this process, there was an unintended change made in the way missing or incomplete data is accounted for.
Statscan collects data from 56,000 households split into six groups. When people don’t respond to the survey, Statscan compensates by inserting plausible values in the holes. It does this in part by checking the new jobs report against the previous month’s, since five of the six groups of respondents are the same from one month to another.
Ironically, changes were made in the system as part of the overall redesign in an effort to reduce errors. But the result was “a higher number of individuals who reported being employed in June, and who did not respond in July, being imputed as either out of the labour force or unemployed in July,” the report states. It wasn’t initially caught because there’s no diagnostic system to detect an error with such an origin.
The report found two key weaknesses led to the error. The analytical team didn’t fully understand the changes to the system, and there weren’t formal reports that looked at how system changes could alter LFS results. Five recommendations were made to prevent similar errors, including better communication and governance.
Arguably most important next step for Statscan will be to develop a more formal protocol to conduct testing, said Claude Julien, director general of the methodology branch at Statscan. This will be implemented across the whole organization.
“I would say as the processing environment and the overall technological environment becomes more complex, we need to have those frameworks to improve and evolve over time as well,” said Mr. Julien, who also helped conduct the review.
Avery Shenfeld, chief economist at CIBC World Markets, said Statscan showed integrity by being upfront with the financial markets that there would be a substantial change to the data, rather than “trying to bury it in a revision down the road.”
The detail in the analysis of the issue could mean Statscan will learn from the error, said Derek Burleton, deputy chief economist at TD Economics. “If the agency implements better procedures and testing, it could reduce the likelihood mistakes will be made in other surveys too,” he said.
One area where many economists would still like to see improvements is the agency’s Survey of Employment, Payrolls and Hours, which looks at 15,000 businesses and payroll data from the Canada Revenue Agency. The report is published two months after the LFS, reducing its market impact. Because of the data collection process, this information comes to market much slower than the corresponding U.S. study does.
“The larger issues are what they don’t report, and don’t have, rather than the methodology behind the surveys they do use,” said Mr. Shenfeld. “We would like the survey of employers done on a more timely basis, like the U.S. has.”
Editor’s Note: An earlier version of this story said that employees of Statscan missed testing their system. In fact, one part of the testing process was missed.