The problem with economic surveys: People aren’t filling them out

Open this photo in gallery:

Statistical agencies are contending with a data-collection issue that boiled over during the pandemic: Increasingly, households and businesses are ignoring surveys that are used to produce key economic figures.

Statistics Canada has struggled to lift the response rate to its Labour Force Survey (LFS), which is filled out by tens of thousands of households every month. In September, the response rate was 70 per cent – much lower than the monthly average of 87 per cent in 2019.

The U.S. Bureau of Labor Statistics (BLS) has encountered a similar trend. Response rates to many of its surveys were gradually declining for years, then worsened during the pandemic. The Current Population Survey, which is used to produce the U.S. unemployment rate, had a response rate of around 90 per cent in 2013. Lately, it’s hovered around 70 per cent.

Despite the smaller sample size, agencies have argued that data quality remains high. But if even more people start tuning them out, the numbers could get distorted – and so, too, our understanding of the economy.

“We’re not happy with the 70-per-cent response rate,” said Vincent Dale, a director-general at Statscan. “But we think our evaluations show that the reliability of the data is very high.”

The LFS response rate tumbled in the early months of COVID-19. For health reasons, Statscan suspended face-to-face interviews, which are a critical way to establish contact with sought-after households. This left two methods for data collection: phone and online interviews.

Every month, Statscan seeks to interview more than 65,000 households, which remain in the LFS sample for six consecutive months. Their responses inform one of Canada’s marquee economic reports, offering a timely reading of labour-market performance, including the monthly change in employment, the unemployment rate and average hourly wages.

Statscan brought back face-to-face interviews in November, 2022. But since then, the response rate has wobbled on either side of 70 per cent – and it certainly hasn’t returned to prepandemic levels.

“People’s willingness to be contacted has changed over the pandemic,” Mr. Dale said.

Nearly two-thirds of LFS interviews were conducted by phone or in person in September. However, because of a change in IT systems in 2020, Statscan can’t say exactly what percentage of recent interviews were solely face to face. (In February, 2020, it was around 20 per cent.)

“We’re not able to report that with precision,” Mr. Dale said.

In the first wave of COVID-19, around three million Canadians lost their jobs, while millions of others retained their roles, but lost the majority of their work hours, based on Statscan’s estimates. But the job losses might have been even larger, according to University of Ottawa professors Pierre Brochu and Jonathan Créchet.

In a paper that was published in Canadian Public Policy last year, they noted that certain segments of the population that were vulnerable to the pandemic’s effect on labour – the young, the less-educated and lower-paid employees, among others – were tougher for Statscan to reach in the initial weeks of COVID-19. As a result, they argued that the decline in employment and labour participation was underestimated during the first wave of infection.

The estimates were “slightly off compared to what we would have gotten in a normal situation,” Prof. Créchet said in an interview.

Last year, the U.S. Census Bureau published an analysis that found income statistics and poverty rates were slightly affected by the slide in response rates over the pandemic. A BLS presentation from 2022 warned that the decline in survey responses was “likely to continue.”

The U.S. data show that businesses are less inclined to fill out surveys than in the past. The Job Openings and Labor Turnover Survey – a key report that documented the “Great Resignation” phenomenon in the U.S. – had a response rate of roughly 66 per cent a decade ago. Now, it’s just above 30 per cent.

A smaller sample size doesn’t mean the data are intrinsically worse. However, there’s a risk of a bias being introduced if respondents differ meaningfully from the non-respondents.

Statscan evaluates the composition of LFS respondents based on various sociodemographic characteristics that include age, education and immigration status. Mr. Dale said Statscan is “quite comfortable” with the sample composition. Moreover, he said the LFS is showing a similar trend for employment as the Survey of Employment, Payrolls and Hours, which is based in part on payroll-deduction records from the Canada Revenue Agency.

“We’re quite confident that the signals coming from LFS are completely reliable,” he said.

Still, the agency is making efforts to reach more Canadians. It sends out introduction letters to selected households to explain the importance of the LFS and encourages them to fill it out. Statscan also follows up with reminders by phone, e-mail and text, among other methods to bolster participation.

Mr. Dale said he expected the response rate to pick up in the coming months. “What we’re doing, month by month, is we’re re-evaluating our strategies to contact people.”