On the one hand, the polls generally did quite well in last night's provincial election in Ontario. On the other hand, they performed quite poorly. This requires some explanation.
During the campaign, several pollsters released two sets of numbers. The first set recorded the support of all eligible voters, the numbers that have traditionally been reported by pollsters. But in the wake of the miss in British Columbia in 2013, many pollsters have endeavoured to estimate what the voting population will look like. And in an election where roughly only half of those eligible to vote actually did, that is an important distinction.
But in every case, the pollsters who employed a likely voter model during the campaign would have been better off had they only reported the traditional numbers among all eligible voters.
In fact, the confusion that the polls injected into the campaign came primarily from these likely voter models. In the end, this confusion was completely unnecessary and any perception that the polls missed the call in Ontario could have been avoided, as the traditional eligible voter numbers proved to be quite reliable.
This is a rather surprising turn of events, and one that is counter-intuitive. With turnout so low, one would expect that estimates of turnout would perform better than polls trying to model 100 per cent of the population. In most elections that would be the case, but it appears that yesterday in Ontario the opinions of people who did not vote lined up relatively closely with the opinions of those who did.
Seven polling firms released numbers in the last week of the campaign. In the final five surveys, which left the field either on June 10 or June 11, the Liberals led in all of them. In the two older surveys, the Liberals were either tied with the PCs or trailed by one point.
In those final five surveys, the Liberals averaged 36.5 per cent support, just 2.1 points lower than their actual tally. The PCs averaged 32.3 per cent support, little different from the 31.3 per cent the party actually took, while the New Democrats averaged 24.2 per cent, virtually identical to their 23.8 per cent result. The Greens averaged 5.2 per cent, but took 4.8 per cent in actuality.
So, in the aggregate, the very last polls of the campaign did an excellent job. They depicted a Liberal victory (though perhaps straddling the minority/majority line, rather than clearing it comfortably), a drop in support for the PCs, and gains for the NDP.
Some individual pollsters did quite well. Angus Reid Global was within 2.6 points for all parties, totalling just 5.7 points of error across the four parties. Abacus Data, polling for Sun News, had a total error of 7.7 points. And EKOS Research, polling for iPolitics, had a total error of 9.3 points, though much of that was due to an under-estimation of NDP support.
If this was the story of the night, we'd commend the pollsters for a job generally well done. The final polls by Forum Research and Ipsos Reid, which had a larger degree of error, still had the Liberals in front.
But that is not the entire story for polling. There was much emphasis put on the likely voter models, deemed to be a necessary tool to avoid another miss like British Columbia.
Four pollsters put out likely voter numbers in their final polls. These varied greatly. Abacus Data and Angus Reid Global suggested a very close race. EKOS Research gave it to the Liberals by six points, while Ipsos Reid gave it to the Tories by that margin. EKOS's likely voter model was the only one to correctly give the Liberals a big win, but it pegged the NDP far too low.
Across these four polls, the Liberals averaged 35.6 per cent, lower than the already-low eligible voter average. The PCs were estimated to have 36 per cent in all four polls, significantly higher than their actual score. The NDP averaged 23.5 per cent, quite close to the result, but that masks the high and low marks of 17 and 30 per cent in these polls.
The four likely voter models averaged an error of 3.5 points per party, compared to an average error of 2.2 points per party for the eligible voter tallies by these same companies.
"We understood the general mood of the electorate but still need to refine how we identify the 50 per cent that will actually vote," said David Coletto, CEO of Abacus Data, in an e-mail. "Measuring public opinion is not easy, but we are always learning and trying to improve. Most important is knowing the limits of the numbers and being transparent with how you get them and interpret them."
Pollsters shouldn't abandon trying to estimate likely turnout. It is a practice that is still in the experimental stage in Canada, though it has been used for some time in the United States. Perhaps it is best to keep working on these models, go back to reporting eligible voter numbers only, and let those who cast a ballot decide elections until the models consistently prove their mettle. But the results last night in Ontario did not represent a miss for the polls – not exactly.
Éric Grenier writes about politics and polls at ThreeHundredEight.com.