Skip to main content
Canada’s most-awarded newsroom for a reason
Enjoy unlimited digital access
$1.99
per week
for 24 weeks
Canada’s most-awarded newsroom for a reason
$1.99
per week
for 24 weeks
// //

The Statistics Canada offices in Ottawa are seen on Tuesday, May 1, 2013.

Sean Kilpatrick/The Canadian Press

The pervasive tone of dissatisfaction among many of Canada's economists to last week's National Household Survey rubbed some readers the wrong way.

They wondered why we need to bother with such incredibly expensive probing of our national populace anyway? Hasn't the science of polling proven that we can take a relatively small sample of Canadians at a relatively small cost, and get very close to the same answers? If polls are good enough for gauging the likely outcomes of something as important as an election, shouldn't they be good enough for, say, telling us how much an average Canadian earns in a year?

These are good questions after the government has just spent a reported $650-million on a massive poll that covered nearly one-third of all Canadian households, and produced a massive amount of data that Statistics Canada considers incredibly valuable – but that many economists still don't think is good enough. Maybe, just maybe, economists are a bunch of ivory-tower grumblers who won't accept a good thing when they see it and aren't spending enough time in the real world.

Story continues below advertisement

Or maybe they are well aware that polls, as much as we have come to rely on them, are very fallible. (Yes, even really big ones.)

Relatively small-sample polling works pretty well on simple questions, but when you seek a lot of increasingly complex and nuanced information (like, say, those contained in the old long-form census, or the NHS), its reliability starts to break down. There are also well-understood biases surrounding who is willing to respond versus who isn't; for example, the top and bottom of the country's socioeconomic spectrum are routinely under-represented in voluntary surveys.

And that value in predicting an election? Lately, not so much. In the Alberta and British Columbia provincial elections of the past two years, the polls famously diverged from the actual results by massive, head-scratching, what-the-heck-were-the-pollsters-drinking margins.

And those election polls – which ask a pretty straightforward question – have errors typically of plus-or-minus 2 to 5 per cent (meaning a range of 4 to 10 per cent), usually 19 times out of 20. Which means that once out of every 20 polls, the results are somewhere outside that range. Where? We have no idea. Statistically speaking, one of of every 20 political polls is just wrong.

Now consider the margin of error in one of Statistics Canada's best-known and closely followed "polls" – its monthly Labour Force Survey, where the country's headlines on job creation and unemployment come from.

Statscan reported last week that Canada had added 59,200 jobs in August – but it also identified its "standard" margin of error as 28,900. What that means is that the job-gain figure is plus-or-minus 28,900, TWO TIMES OUT OF THREE. Which means one in three times, the survey is outside this sizable range. You want to ramp that up to 19 times out of 20, like in an election poll? Then you DOUBLE the standard error. Which means that when you see "Canadian jobs worse 59,200 in August," what the polling error is actually telling us is that there's a 95-per-cent likelihood that Canada gained somewhere between 1,400 and 117,000 jobs last month. And there's a 5-per-cent chance that it was some other completely different number outside that range – perhaps even a big job loss.

This is why we bother with a national census, or the National Household Survey. Every few years, it's important that policy makers get as full and complete a set of data on the country as they possibly can; they don't want to making long-term, multi-billion-dollar government program decisions based on potentially flawed, skewed, inaccurate or flat-out wrong data.

Story continues below advertisement

Of course, Statscan also does many smaller samplings to gauge the state of many elements of Canadian society and the economy as we go along, and without doubt, they are useful and relatively cost-effective. But small samples are imperfect.

As an illustration, I present a statistical argument that one reader raised as evidence that we don't need big polls: It's known as "The Birthday Paradox."

You hire a pollster to start calling people in search of two people with the same birthday. You might think it would take, on average, 366 calls to hit on a match (that's the total number of possible birthdays, and we figure on a pretty even distribution). However, we discover that statistically, there's a 50-per-cent chance that the pollster will find two people with the same birthday after just 23 calls; once he has talked with 57 people, the probability of a match goes up to 99 per cent. So, it turns out, it's a complete waste of time and resources to talk with 366 people. (There are some quirky statistical reasons why this finding is misleading – after all, it's not called a "paradox" for nothing – but I'm assured that it does work.)

So my pollster stops making calls after 57 people (or fewer), since he almost certainly found a match, and saved me a tonne of money. But what do my statistics suggest? That 3.5 per cent (or more) of Canadians have the same birthday. Intuitively, I know that's wrong; simple logic suggests that it's more like 0.3 per cent (i.e one out of 365). Yet the data from my small-sample poll tells me something wildly different.

That's not so useful, is it?

Your Globe

Build your personal news feed

  1. Follow topics and authors relevant to your reading interests.
  2. Check your Following feed daily, and never miss an article. Access your Following feed from your account menu at the top right corner of every page.

Follow the author of this article:

View more suggestions in Following Read more about following topics and authors
Report an error Editorial code of conduct
Due to technical reasons, we have temporarily removed commenting from our articles. We hope to have this fixed soon. Thank you for your patience. If you are looking to give feedback on our new site, please send it along to feedback@globeandmail.com. If you want to write a letter to the editor, please forward to letters@globeandmail.com.

Welcome to The Globe and Mail’s comment community. This is a space where subscribers can engage with each other and Globe staff. Non-subscribers can read and sort comments but will not be able to engage with them in any way. Click here to subscribe.

If you would like to write a letter to the editor, please forward it to letters@globeandmail.com. Readers can also interact with The Globe on Facebook and Twitter .

Welcome to The Globe and Mail’s comment community. This is a space where subscribers can engage with each other and Globe staff. Non-subscribers can read and sort comments but will not be able to engage with them in any way. Click here to subscribe.

If you would like to write a letter to the editor, please forward it to letters@globeandmail.com. Readers can also interact with The Globe on Facebook and Twitter .

Welcome to The Globe and Mail’s comment community. This is a space where subscribers can engage with each other and Globe staff.

We aim to create a safe and valuable space for discussion and debate. That means:

  • Treat others as you wish to be treated
  • Criticize ideas, not people
  • Stay on topic
  • Avoid the use of toxic and offensive language
  • Flag bad behaviour

If you do not see your comment posted immediately, it is being reviewed by the moderation team and may appear shortly, generally within an hour.

We aim to have all comments reviewed in a timely manner.

Comments that violate our community guidelines will not be posted.

UPDATED: Read our community guidelines here

Discussion loading ...

To view this site properly, enable cookies in your browser. Read our privacy policy to learn more.
How to enable cookies