Paul Chiasson
Many federal election news stories have been generated by one party or the other going up or down in the polls by one or two per cent. The stories almost always note that polls have margins of error.
For example the recent Nanos poll is based on about 1000 respondents, giving a statistical margin of error of 3.1, percentage points, nineteen times out of twenty.
What is not mentioned in the reporting is that changes between polls have about a 40 per cent higher margin of error than individual polls themselves.
So even if a party is up say 3.5 percentage points comparing a new poll with a previous poll, if each poll had a margin of error of 3.1 percentage points, the 3.5 percentage point increase should be compared to a margin for error of about 4.4 percentage points. There is a reasonable chance that the party's vote intention share in the total population did not change at all: all that happened was that the pollsters randomly happened to choose more of the party's supporters in the second poll.
The margins of errors for changes in leads can be twice as large again. If a party is leading by 5 percentage points in one poll and then by 9 percentage points in the next poll, the margin of error around that 4 percentage point gain could be over 8 percentage points. While probably the lead increased, there is still a significant chance that the lead decreased.
Moreover, the official error calculation assumes that the sample of respondents is random and that responses are accurate: it only takes into account the error associated with trying to make inference from a sample of 1000 people as to what 15 million voters might be thinking.
But there are probably additional errors, especially when comparing polls from different pollsters. Has anyone else wondered why the Greens consistently score better on the Ekos poll (9.3 per cent over the last two weeks) than the Nanos poll (3.75 per cent over the same period)? At first I wondered whether it was just that respondents hear "Ekos" which puts "Eco" in their minds, but compared to the other polls, Nanos is just as much on the low side as Ekos is on the high side. I don't know the answer, but it is a reminder that polling techniques likely matter.
Most everyone takes polls with a grain of salt. When considering changes from an individual poll to the next, I take extra grains.
Michael Veall is an economics professor at McMaster University
Follow Economy Lab on twitter