What is going wrong with the polls?
Last week, after trailing in the polls right up to election day, the British Columbia Liberals won another mandate – a provincial election that the media and public are calling a "surprising" result. Now, we are amidst a host of pollster mea culpas and claims by some firms that they got it "less wrong than the others." Herein lays a new problem for the media, politicians and the public: our faith in the accuracy of political polling.
First, let me disclose that I conduct public opinion research as part of my business, and have done political polling in the past. Personally, I have long regarded this collapse of faith as a looming problem, one that has become very serious over the last two years.
Political polling is traditionally a test of the accuracy of a public-opinion research firm. It is considered a loss leader (typically offered below cost), with the expectation that an accurate poll result will build a firm's reputation and attract new and more profitable business.
Over the last decade this premise has changed dramatically. With the rise of the software-based polling methods – notably online polling using Internet panels of self-selected respondents, as well as interactive voice response systems (typically referred to as "robodials" by the public) – the cost of entry for new methods and firms has never been cheaper. Gone are the days of excellent response rates to telephone (landline) polls. And gone are the days of predictably engaging the public to garner its political inclinations.
We are in a new world – low voter turnouts, multiple communication technologies, social-media platforms, and the use by parties of geo-demographic targeting and sophisticated voter identification methods to find supporters. These have dramatically affected the political polling business, and pollsters have been slow to adjust or have failed to evolve their skills.
There is that old saying – "Fast, cheap and good. Pick two" – which is truly applicable here. While corporations typically choose a combination of fast or cheap with good, media outlets in recent years have always opted for fast and cheap. The business model of polls and the media has evolved. Media are currently either cash-strapped or losing money, and thus, in most cases, either do not pay for political polling or pay for access to polls already conducted.
There is the additional dimension of politicos and the media's obsession with the "horse race." Many column inches are taken up with the analysis of poll results and insights from pollsters (some of you may include this article in that category as well). While these stories do capture the pulse of an election, they do not take into account the overall election ecosystem and the body politic.
Let's delve deeper.
After the 2012 U.S. national election, Nate Sliver, most likely the most famous political statistician at this moment, published an eye-opening analysis of all the polling data-collection methodologies and pollster accuracy rates. The findings were revealing – fast and cheap methods had larger respondent biases (by supporters of specific political parties) and were less accurate. Surprisingly, the best-performing poll was the Columbus Dispatch's old-school mail survey. Over all, live telephone-operator and Internet-panel polls performed significantly better than robodials. These methods were better at establishing a more population-representative sample that captured the diversity of opinion and voting behaviour. However, they are also significantly more expensive than the cheap-to-operate, large-sample, conducted-overnight robodials. Clearly there is a tradeoff here.
This leads to three dimensions of polling itself:
1. The media and the parties are asking different questions.
Polling, in its cheapest form, focuses on the horse race. But elections are more than that. They are tests of political parties' brands, the public's confidence in the economy and their governments' stewardship, and societal trust. Quality polling captures these elements, and how they wax and wane during the writ period. Quality polling also entails more in-depth, statistical analysis that addresses aspects such as tests of correlation and voter segmentation – aspects that Nate Silver and his more methodical contemporaries embrace.
2. Political war rooms use a variety of tools the media don't.
There is an inherent misalignment between pollsters and party war rooms. Pollsters have polls. War rooms have polls, plus social-media monitoring platforms, feedback from their ground network, content analysis of media coverage, text analysis of editorials and public comments, as well as voter-identification systems. Pollsters mostly ignore this latter element, but parties are investing heavily in it. The Conservative Party of Canada uses CIMS, while others are using Obama's platform of choice, NationBuilder. These platforms are meant to address a question rarely considered in the media: What is a party's secure and confirmed vote? Polls are not designed to capture this data, but voter identification is playing a larger role in election outcomes. Some parties are clearly better at getting their vote mobilized and to the polls on Election Day. The Conservative Party of Canada's 2012 federal majority is a testament to this.
Further, data triangulation – finding the best insights across multiple sources - has always been a skill amongst the best war room teams. It is no surprise that data scientists – those with triangulation, interpretation and communication skills – are much sought after by political parties. Their talents are becoming more useful than those of the traditional party pollster.
3. There is a consistent misalignment of voter intentions and voter turnout.
In most cases, answering a poll is not akin to actually voting. Polling exposes social desirability bias – I say I vote because it is the right thing to say, even if I don't actually vote. Saying you want change and voting for change are independent events. This was evident in all of these "surprise" results. In my opinion, the real metrics that matter relate to the committed or intending voter. These are poll respondents who have a history of voting (themselves and in their family tradition) and intend to vote on election day. Based on my analysis, while it may result in a small respondent base with a higher margin of error, this group has been a better predictor of voter turnout. Observing this metric within the context of the recent BC and Alberta elections, there were warning signs that things turned for the eventual winner earlier than what most pollsters believed.
The current business model of the media overrides any quick resolution of the "fast and cheap" polling problem. It does, however, exacerbate the biggest problem for pollsters – one facing political parties and democracy itself: low voter turnout. B.C. is flirting with the 50 per cent floor, and Alberta saw turnout drop to 41 per cent in 2008. There is much research into this, but no matter what, the ranks of non-voters are growing, and no amount of suspect polling is going to answer these questions.
Brian Singh, the president of Zinc Research, is a political consultant based in Calgary.