Critics who protested the federal government's decision to cancel the long-form census have had their worst fears confirmed with the release this week of data from the survey that has replaced it. The National Household Survey is weakest and most unreliable in its data on low-income earners, which is exactly the danger that many observers foretold when the Harper government announced the cancellation of the long-form census in 2010.
Statistics Canada acknowledged the survey's problems prominently in Wednesday's release, warning readers that new data for low-income earners "show markedly different trends than those derived from other surveys and administrative data." This is a technical way of saying the data don't appear to fit with other information on much the same topics, collected from other sources.
Statscan also warns that, because of methodology changes, the 2011 survey results cannot be compared with 2006 census data to develop critical comparative statistics about changes in income. It means that despite spending over $650-million to collect data from Canadians – a cost increase of at least 15 per cent over the last census, in 2006 – this survey could not offer insight into such basic questions as whether the poor are getting poorer, or whether average incomes are rising or stagnating.
The problem is that a census is a mandatory collection of data, while the new survey is entirely voluntary. The 2006 census had a response rate of 93.5 per cent, while the new survey had a response rate of 68.6 per cent. The difference wouldn't be critical if everyone were equally likely or unlikely to respond, but this has never been the case. Low-income earners are typically less prone to answer voluntary surveys, so results tend to be skewed by a sample of respondents who are not representative of the whole population. The super-rich are no better, and economists have also noted they do not appear to have completed the new survey in significant numbers compared with their relative weight in the population.
At both income extremes, it is unclear whether the survey data are reliable, with Statscan itself raising red flags about the low-income data. Income information is available from other sources, of course, but the point of doing an expensive national survey is to get comprehensive data that offer the best information to policy-makers. This has not happened. The solution is to acknowledge the weaknesses and convert the survey back into a proper national census.