Skip to main content

The Statistics Canada offices in Ottawa are seen on Tuesday, May 1, 2013.

Sean Kilpatrick/THE CANADIAN PRESS

Canada's National Household Survey on incomes produced flawed data with harmful implications for public policy, according to a range of researchers and statistics experts who have sifted through the numbers.

Consultants, urban planners and health policy experts say the data quality is worse than they'd expected, masking key shifts in income inequality and poverty in the country. The blurred picture has also left them unable to track trends over time. And they question whether the higher cost to produce data of worse quality – at $22-million extra for the survey for a total census cost of $652-million – amounts to wasted taxpayers' dollars.

Statistics Canada developed the new voluntary survey after the federal government cancelled the mandatory long-form census in 2010. Some call the data from the survey "dangerous" because lower response rates obscure who is faring better and who is worse off – which could lead to misguided policy decisions in the years ahead.

Story continues below advertisement

Voluntary surveys typically garner weaker response rates from those at the higher and lower ends of the income spectrum. And response rates to the National Household Survey show much lower responses in many mid-sized cities, smaller communities and rural areas.

The 2006 mandatory long form, which required everyone who received a questionnaire to fill it out, had a response rate of 93.5 per cent. The NHS was voluntary and had a much lower response rate of 68.6 per cent. A less representative sample makes it much more difficult to draw unbiased conclusions.

Many who rely on the data plan to discard using it altogether.

"The Auditor-General should get on this one," said Thomas Lemieux, professor at the University of British Columbia and president of the Canadian Economics Association.

He says he will stay clear of the NHS data in assessing long-term trends on income inequality. "I had a secret hope that because Statscan has lots of information to fill the hole that they would be able to create still a workable, good-quality product. My heart fell when I saw that Statscan said they're not going to compare to 2006...I didn't think it would be that bad. "

Munir Sheikh, Statistics Canada's former chief statistician who resigned in 2010 over the decision to scrap the census, said the effort has been a waste of money.

"The irony is, we've spent more money compared to a census to get data which is largely useless. Why anyone would want to do this is beyond me. Why would you spend $600-million for this?" said Mr. Sheikh, who is now an adjunct professor at Queen's University.

Story continues below advertisement

In Winnipeg, Dr. Patricia Martens, director of the Manitoba Centre for Health Policy, says flawed or missing data – that under-represents, for example, the number of poor children in her province – could lead to misaligned funding for early childhood education programs or immunization campaigns.

"I worry we've left out the most vulnerable people, so we might make erroneous statement like the income gap is shrinking when it's not," said Dr. Martens, who is also a professor at the University of Manitoba's faculty of medicine and member of the Order of Canada.

Her mistrust of the data is such that her team plans to ditch the NHS altogether, and instead rely on the older, mandatory 2006 census.

Statscan has long been regarded as one of the top statistical agencies in the world and many held out hope its respected analysts would find a way to make the data comparable with previous years. But the agency bluntly said in its Sept. 11 release people should "use caution" in comparing the NHS income data with other surveys or censuses due to the methodology change.

David Hulchanski and a team at the University of Toronto have tested the NHS income data. They have found it doesn't align with annual tax-filer data, with "wonky" results in many areas – in particular showing rising incomes among poor people and falling ones among the rich.

"We're concluding it pretty much is garbage," said Prof. Hulchanski, who co-directs data analysis of the Neighbourhood Change Research Partnership based at U of T.

Story continues below advertisement

They are concerned the data will be used for partisan purposes and make it seem like Canada is becoming a more equal society "when the opposite is happening."

In Calgary, consultant Robert Gerst is urging his clients – municipalities, provinces and companies in the oil-and-gas sector – not to use the new data. "It's dangerous," he said.

Planners in his city have already started using the survey to help determine where to place new bus routes – though faulty data could mean empty seats and millions of dollars wasted, he said. "It's a mess because it's not that the data is wrong. It's unreliable – we don't know whether it's right or wrong, we have no clue. And there's no scientific way to tell."

Not everyone is ditching the survey. Doug Norris, chief demographer at Environics Analytics in Ottawa, says though he won't use it for long-term comparisons, it still provides a useful snapshot of how incomes in some cities or provinces compare with others. The problem may also be less acute for the private sector because many businesses rely on their own market surveys to make decisions.

Statscan says each user should determine for themselves if the NHS income information meets their particular needs.

"The fit for use depends on the particular analysis done," Statscan said in an e-mailed statement to The Globe. The new survey "is a very rich source of information and meets the requirements of most users."

Story continues below advertisement

The federal government hasn't yet said what it plans to do for the 2016 census. Mr. Sheikh recommends making Statistics Canada an independent agency from the government, similar in structure to the Canada Revenue Agency. Dr. Martens in Winnipeg wants to see the return of the mandatory long-form census. And the University of Toronto professors recommend the next census be restored to the "non-politicized, non-partisan scientific methodology that existed before" the NHS.

Industry Canada, which oversees Statscan, told The Globe a review is still under way for the next census. "We continue to look at ways to improve the next census cycle while protecting Canadians' privacy."

Report an error Editorial code of conduct
Comments

Welcome to The Globe and Mail’s comment community. This is a space where subscribers can engage with each other and Globe staff. Non-subscribers can read and sort comments but will not be able to engage with them in any way. Click here to subscribe.

If you would like to write a letter to the editor, please forward it to letters@globeandmail.com. Readers can also interact with The Globe on Facebook and Twitter .

Welcome to The Globe and Mail’s comment community. This is a space where subscribers can engage with each other and Globe staff. Non-subscribers can read and sort comments but will not be able to engage with them in any way. Click here to subscribe.

If you would like to write a letter to the editor, please forward it to letters@globeandmail.com. Readers can also interact with The Globe on Facebook and Twitter .

Welcome to The Globe and Mail’s comment community. This is a space where subscribers can engage with each other and Globe staff.

We aim to create a safe and valuable space for discussion and debate. That means:

  • All comments will be reviewed by one or more moderators before being posted to the site. This should only take a few moments.
  • Treat others as you wish to be treated
  • Criticize ideas, not people
  • Stay on topic
  • Avoid the use of toxic and offensive language
  • Flag bad behaviour

Comments that violate our community guidelines will be removed. Commenters who repeatedly violate community guidelines may be suspended, causing them to temporarily lose their ability to engage with comments.

Read our community guidelines here

Discussion loading ...

Due to technical reasons, we have temporarily removed commenting from our articles. We hope to have this fixed soon. Thank you for your patience. If you are looking to give feedback on our new site, please send it along to feedback@globeandmail.com. If you want to write a letter to the editor, please forward to letters@globeandmail.com.
Cannabis pro newsletter