Jeffrey S. Rosenthal is a professor of statistics at the University of Toronto whose books include Knock on Wood: Luck, Chance, and the Meaning of Everything.
The U.S. presidential election gripped us all. Razor-thin margins. Four days of uncertainty about the outcome.
It wasn’t supposed to be this way. Joe Biden was expected to defeat Donald Trump easily, in a rout. How did we know? The polls told us so!
Ah yes, the polls. As a statistics professor in the public eye, I am asked about public opinion polls all the time. Can we trust them? Are they accurate? How did they go so wrong? Why should we ever bother with them again?
I even hear this from other statisticians. After Mr. Trump’s surprise 2016 victory, one statistician told me that he would like to talk to pollsters “after a few drinks” to learn what was really going on. This week, another statistician asked what I could possibly say to the media about polls, whose recent inaccuracies had completely shaken her confidence.
So, have opinion polls been revealed, once and for all, to be shams, unworthy of our attention? No!
For one thing, the polls weren’t as far off as many people think. Early partial counts showed Mr. Trump massively outperforming expectations. But the final counts, which is what polls try to predict, will have Mr. Biden up by several percentage points in the popular vote. This is not as high as the eight to 10 points the polls predicted, but not so far off either.
Furthermore, he will end up winning most of the states he was expected to – though still missing a few (such as Florida), and nearly missing some supposedly “easy” ones (such as Wisconsin).
Also, emotions can cloud our judgment. Most Canadians, and most statisticians, strongly preferred Mr. Biden to Mr. Trump, and were very upset that he underperformed. If Mr. Biden had, instead, overperformed and won the election twice as easily as predicted, they probably wouldn’t have complained nearly as much.
But the real issue is that polling is much harder than most people realize. I am often asked, “How can we predict millions of votes, based on just a few thousand random samples?” Actually, that part is easy – random samples are a kind of repeated experiment, for which simple statistical analysis can derive estimates, forecasts, probabilities and margins of error with ease.
The problem is, poll samples are not random. Oh, the pollsters try. They phone people randomly, but most people do not answer. The dirty little secret of polling is that response rates are very small: fewer than one person in 10 will actually pick up the phone, agree to talk to the pollster and reveal voting preferences. And the rest? Why do they not answer, and how will they vote? We’re not sure – they never speak to us.
The pollsters are left with the challenge of forecasting how everyone will vote, based on responses from the small, non-representative minority they can reach. This task is essentially impossible – if it were given as a question on a statistics exam, the students would all complain that it was unfair.
Pollsters soldier on, reweighting their samples using demographic information such as race and age and education to try to mimic the full population as best as they can. Sometimes they succeed very well – polls accurately predicted the U.S. presidential elections of 2008 and 2012, the U.S. midterm elections of 2018, the Canadian federal election of 2019, and several recent Canadian provincial elections.
Other times the polls miss out, like underestimating Mr. Trump’s support in the 2016 and 2020 U.S. presidential elections (though still not as badly as many people think). Why? Perhaps some of Mr. Trump’s supporters were too “shy” to admit their preference, and others were less likely to talk to “elite” pollsters, and some were harder to reach owing to work and other responsibilities – nobody knows for sure.
Some pollsters have given up on random samples entirely. They use alternative methods such as online panels, computer analysis of posted tweets (the Ottawa-based company Advanced Symbolics had great success with this in the 2019 Canadian federal election, but similarly missed the 2020 U.S. presidential election), or even intercepting and surveying people while they browse the web (Toronto-based RIWI used this to correctly predict a tighter 2020 race). They are all chasing the same elusive dream, to figure out how everyone is thinking based on samples neither random nor complete.
So, should we give up on polls forever? Hardly. Without poll results, election commentators would have few insights beyond whatever tidbits they overheard in bars and gatherings. Candidates would have no idea how to prioritize their campaign plans. Voters would have no clue about which races were close, and how to vote strategically to achieve their desired outcome.
Polls are here to stay, flaws and all. And really, is that such a terrible thing? If pollsters could predict every election with perfect accuracy, campaigns would be irrelevant, citizens would feel superfluous and there would be no reason to hold the vote. The limitations of polls make them sometimes very accurate and sometimes less so. We should trust them somewhat, but not completely, and they can never replace the actual election that follows.
In the end, election polls are like movie trailers. They give you a pretty good sense of what to expect. But if you want to know what really happens, you still have to watch the movie.
Keep your Opinions sharp and informed. Get the Opinion newsletter. Sign up today.