Skip to main content

The Globe and Mail

Take poll changes with two grains of salt

Paul Chiasson/Paul Chiasson/The Canadian Press

Many federal election news stories have been generated by one party or the other going up or down in the polls by one or two per cent. The stories almost always note that polls have margins of error.

For example the recent Nanos poll is based on about 1000 respondents, giving a statistical margin of error of 3.1, percentage points, nineteen times out of twenty.

What is not mentioned in the reporting is that changes between polls have about a 40 per cent higher margin of error than individual polls themselves.

Story continues below advertisement

So even if a party is up say 3.5 percentage points comparing a new poll with a previous poll, if each poll had a margin of error of 3.1 percentage points, the 3.5 percentage point increase should be compared to a margin for error of about 4.4 percentage points. There is a reasonable chance that the party's vote intention share in the total population did not change at all: all that happened was that the pollsters randomly happened to choose more of the party's supporters in the second poll.

The margins of errors for changes in leads can be twice as large again. If a party is leading by 5 percentage points in one poll and then by 9 percentage points in the next poll, the margin of error around that 4 percentage point gain could be over 8 percentage points. While probably the lead increased, there is still a significant chance that the lead decreased.

Moreover, the official error calculation assumes that the sample of respondents is random and that responses are accurate: it only takes into account the error associated with trying to make inference from a sample of 1000 people as to what 15 million voters might be thinking.

But there are probably additional errors, especially when comparing polls from different pollsters. Has anyone else wondered why the Greens consistently score better on the Ekos poll (9.3 per cent over the last two weeks) than the Nanos poll (3.75 per cent over the same period)? At first I wondered whether it was just that respondents hear "Ekos" which puts "Eco" in their minds, but compared to the other polls, Nanos is just as much on the low side as Ekos is on the high side. I don't know the answer, but it is a reminder that polling techniques likely matter.

Most everyone takes polls with a grain of salt. When considering changes from an individual poll to the next, I take extra grains.

Michael Veall is an economics professor at McMaster University

Follow Economy Lab on twitter

Story continues below advertisement

Report an error Editorial code of conduct
As of December 20, 2017, we have temporarily removed commenting from our articles. We hope to have this resolved by the end of January 2018. Thank you for your patience. If you are looking to give feedback on our new site, please send it along to If you want to write a letter to the editor, please forward to