Statistics Canada chief defends data as quality concerns persist

Share

More researchers are raising concerns about the quality of data from Statistics Canada's national household survey, even as the agency's chief statistician says the results have exceeded his expectations.

Back in 2013, Wayne Smith called the NHS – conducted for the first time in 2011 after the federal government cancelled the mandatory long-form census – a "success." Now, he says the quality has turned out better than he'd expected.

"My view is more positive now than it was, based on everything that we've learned about the data subsequently," Mr. Smith said in an interview, adding that he has trouble understanding why people who use the data are so "skittish" about it.

The federal government cancelled the long form in 2010 in favour of a voluntary household survey, saying it was too intrusive and Canadians shouldn't have to go to jail for not filling it out. Many had warned that the switch – at an extra cost of $22-million – would mean lower response rates and a poorer basis upon which to develop social and economic policies. The response rate for the 2011 NHS was 69 per cent compared with 94 per cent for the 2006 mandatory long-form census.

Mr. Smith's views are at odds with many users of the data – from economists to urban planners and social policy researchers – who are concerned about the reliability of the survey, particularly at the local level. This week, the city council of Burnaby, B.C. passed a motion asking for the reinstatement of the mandatory long form, saying municipal governments are "particularly affected by this change given that the data is increasingly unreliable at the local neighbourhood levels."

A number of experts, such as Charles Beach, professor emeritus in economics at Queen's University, say Canadian taxpayers have wound up paying more to get data of worse quality.

The next census year is 2016 and the current plan is to conduct another voluntary household survey and a mandatory short-form census of the population.

In the last effort, several vulnerable groups – recent immigrants, aboriginal people off reserve and low-income people – were considered at risk of being under counted in the NHS, as they are typically less likely to fill out voluntary surveys.

Mr. Smith said that many of those data-quality concerns "weren't borne out" and there's no evidence to suggest they were significantly under counted. In fact, he said there "were a lot more First Nations off reserve than there should have been," which may reflect more pride in their heritage and thus more likelihood to self identify.

He pointed out that no survey has a 100-per-cent response rate. "Why people are suddenly so skittish about using this survey with a massive sample response, of [about] 2.5 million households … and a response rate that is rarely seen in any survey conducted in Canada by anybody other than Statistics Canada, why they would get skittish about that is not obvious."

A large sample size is not an adequate counter to low and uneven response rates, Prof. Beach said. "It just means that you have a lot of biased data."

He is still concerned over response rates, particularly from low- and high-income households, that he said have made it more difficult to track and compare poverty rates and income inequality measures over time.

As well, there is a lack of historical comparability with previous census results; another worry is weak coverage of income and labour market trends at local levels.

The 2011 census cost $652-million, plus an additional $22-million to conduct the NHS. The budget for the next effort is expected to be tabled later this month. The general tendency is for census costs to increase, Mr. Smith said, because the number of households grow (by a million as of next year compared with the last census) and because of inflation for such things as printing, postage and wages. About 35,000 people will be hired.

In the end, the "simple fact that taxpayers are paying more [for the voluntary NHS] and getting less, in terms of reliable and historically comparable results – not what I would call a good deal," Prof. Beach said.

While his confidence in the NHS income data is "higher," Mr. Smith said the agency is "still in a situation where we're saying use with caution." And although caution is warranted in analyzing small-area level data,"when you start to aggregate it, the problem largely goes away."

Lessons learned from the 2011 effort will help the agency try to make the 2016 estimates "even more solid than they were," Mr. Smith said.

Mr. Smith said his confidence in the NHS income data, specifically, "is actually rising," partly because the NHS low-income data seems to fit with information it has gotten from its more recent Canadian income survey. "I understand, if somebody says there was a degradation of quality -- yes. I don't dispute that, relative to the 2006 census, due to the change in response rates. But to say that on that basis we should just throw out the data because it's no good, is also nonsense."

Statscan's former chief statistician Munir Sheikh resigned in 2010, saying a voluntary survey cannot serve as a replacement for a mandatory census. Mr. Smith, appointed in 2011, wouldn't say what format he would personally prefer for the next census. But he did say it is likely that Canada - along with other developed nations - will increasingly rely on administrative data (such as tax records) in the coming years.