Experts debate how much National Household Survey statistics count

Share

The Harper government's controversial decision to shield Canadians from intrusive census-takers is coming home to roost this week.

It was nearly three years ago that the Conservatives axed the mandatory long-form census, a list of about 50 questions that one-fifth of Canadian households were previously compelled to answer.

Declaring that they didn't want Canadians to be forced by law to divulge exhaustive details about their lives, the Conservatives left it to Statistics Canada to come up with a voluntary alternative – one that nobody is required to complete.

The result was the 2011 National Household Survey, from which the first findings are being released Wednesday. It deals with more probing matters than the short-form census, which is still mandatory and asks Canadians 10 basic questions such as birth date and languages spoken.

The inaugural National Household Survey will offer a snapshot of the lives of Canadians, but one that will be less detailed, and more spotty, than what Statistics Canada used to be able to generate.

At the 30,000-foot level it will be difficult to spot the difference, but as users drill down to smaller communities or small geographical areas they will likely discover the findings are fuzzier, or even missing.

"This will not have the detail or the precision of the traditional long-form census," said Ian McKinnon, chair of the National Statistics Council, which advises Canada's chief statistician. "For small groups and small areas, it will be harder to get a clear view of Canada."

This will leave urban planners, market researchers, school boards and other policymakers with holes in their knowledge as they try to make plans for everything from bus routes to store locations to where to situate new schools or set up services for immigrants.

Prime Minister Stephen Harper's decision to curb Statscan's reach has been seized by critics as proof that the Conservatives are trying to downgrade the importance of evidence-based decision-making. The death of the long-form census is now part of the opposition narrative that frames the them as anti-science, along with their withdrawal from the Kyoto accord on global warming and their muzzling of scientists.

But opinion is mixed on how big a loss the long-form census represents.

Pollster Darrell Bricker, who surveys Canadians and foreigners for a living, says the fuss is overblown.

"It's all based on a myth," said Mr. Bricker, chief executive officer of Ipsos Public Affairs. "The myth [is] that the long-form census was perfect. It wasn't."

Critics say some groups – aboriginals, the poor, as well as young adult males and older people – will be less accurately counted because they are typically less likely to respond to voluntary surveys.

"The truth is that the people they are most worried about losing weren't responding to the surveys anyway," Mr. Bricker said. "They had really low response rates among aboriginals. Poor people were always hard to reach."

Still, even Statistics Canada itself is blunt about how the change has affected the quality of data that will be gathered every five years.

"We have never previously conducted a survey on the scale of the voluntary national household survey, nor are we aware of any other country that has," the agency says on its website.

The new methodology has been introduced "relatively rapidly with limited testing." The effectiveness of its strategy to offset increased inaccuracy "is largely unknown." And for these reasons, "it is difficult to anticipate the quality level of the final outcome," the agency warns on its website.

Mr. Bricker says he's never regarded the long-form census "as a biblical document," noting that in 2001 more than 20,000 Canadians registered their religion as Jedi, the fictional philosophy from the Star Wars movie saga.

Ivan Fellegi, a former chief statistician who led the agency between 1985 and 2008, says the long-form census is still a better representation of the Canadian population than the National Household Survey.

"Yes, not absolutely everyone may have told us the truth, but I trust Canadians: most did. Why would they lie?"

Some of the data quality from the survey will be good and some will be bad, but it will be tough to distinguish one from the other, Mr. Fellegi said.

"Great caution is what I would advocate," he said.

In practice, that means people should try to verify surprising results by comparing against other sources of information to confirm whether it's a new trend or an error, he said.

Mr. Fellegi is confident that the agency has done all it can to generate the best products given the limitations placed on it.

But the detailed data, such as trends at the neighbourhood level, warrants the most caution – as do comparisons with prior years.

If any of the results yield surprises, "my advice is to be very, very cautious and make sure you understand why."

For most ordinary users, Mr. McKinnon says, there'll be lots of useful data.

Finn Poschman, vice-president of research at the C.D. Howe Insitute, predicts federal and provincial governments will find enough detail to help inform "big-picture questions" where the trends and broad analysis are key.

Mr. Bricker says policymakers and researchers who need more detail should consider conducting their own surveys and market research. The NHS, to be conducted every half decade along with the short-form census, "is a really static method for interviewing members of a population."

"In the world of big data there's tonnes and tonnes of information out there. We have more data than we know what to do ... we have more access to data than we've ever had."

Paul Jacobson, Toronto-based economics consultant and vice-president of the Canadian Association for Business Economics, doesn't think a marketing firm could augment the data no longer supplied by Statscan.

The best way to improve the data would be to boost the frequency at which this survey is conducted, he added – to every six months to a year.

Then, after five or six years, "I'd be a little more comfortable with it."

Critics say that traditional response rates to market-research firm surveys are so low that the cost to produce better results than the NHS might be prohibitive.

Karen Leibovici is president of the Federation of Canadian Municipalities, which relies heavily on census data to help determine growth patterns and city planning. She says it's unrealistic to think cities would be able to hire marketing firms to fill in the gaps in the data. "Municipalities are [too] strapped for financial resources to go out and hire marketing firms – especially the smaller municipalities."

"The question is, can we fill in the blanks, and what will it take to fill in the blanks."

The accuracy problems with the National Household Survey will only increase over time, experts say. That will put pressure on the federal government to reverse the decision taken by the Harper Conservatives, they predict.

In the short term, researchers can use the 2006 long-form census as a benchmark to help cross-check the NHS's findings, statisticians say.

But as patterns of settlement, immigration and population growth shift, that 2006 reference point will become less relevant.

That means the information on smaller communities, ethnic groups and tinier geographical areas will become fuzzier over time.

If the Conservatives won't change their minds, Mr. Fellegi recommends adding a few key questions to the still-mandatory short-form census so that a bit more data is collected from all households.

A few extra questions on income levels, education, immigration and aboriginal status would help paint a better picture of "vulnerable groups," he said.