Donna Dasko is former senior vice-president of Environics Research Group; she was co-author of the Globe-Environics poll (1984-1989) and directed CBC/Environics polls (1993-2008).
----------------------
In the past week, the pollsters reached a consensus on the outcome of Monday's federal election: Most firms had Justin Trudeau and his Liberals with a lead that would likely win the party a minority government. The consensus put the Conservatives in second place, with the once-resurgent New Democrats in third. Media commentary follows the polls closely; almost everything written or spoken about the horse race accepted this narrative.
But what if the polls are wrong? In the 2013 British Columbia election, every one of eight opinion polls conducted within one week of election day showed the New Democrats poised to win a majority – but the Liberals won a majority instead.
Similarly, in the 2012 Alberta election, all seven opinion polls had the Wildrose Party with a lead, but the Conservatives ended up winning a majority. In the 2011 federal election, while pollsters documented the rise of the New Democrats to second place, almost none predicted the Conservative majority.
The polling industry has reached a "crisis point" according to Darrell Bricker, chief executive officer of Ipsos, one of Canada's leading survey firms. The New York Times, pointing to polling disasters in Britain, Israel and the 2014 U.S. midterm elections, recently echoed that sentiment.
The dubious reliability of current opinion poll results can be traced to a host of new technologies and the declining viability of traditional surveying methods. The main problem boils down to changes in sampling – how pollsters find people to survey and whether people answer the polls.
The random sample is the gold standard of polling: If a sample is chosen randomly from a population, that sample will have the characteristics of that population, within a margin. When I began in the industry in the 1980s, we were able to get excellent, if not perfect, random samples through random-digit dialling. Almost every Canadian household had a landline phone, creating potential access to almost the entire population. People were interested in answering our surveys, especially about politics and public affairs, and response rates were high. We had ideal conditions for practising our trade.
The first warning signs came in the decade of the nineties, when response rates started to drop. The following decades saw the growth of recorded messages, call display and cellphones, and further declines in response rates as the novelty of being wooed by pollsters wore off. I recall response rates of up to 80 per cent in the phone polls we did in the eighties, then dropping to the 40-per-cent to 50-per-cent range a decade later. Now, traditional telephone surveys (with live interviewers) are getting response rates of about 10 per cent or less.
The growth of cellphones has created a new sampling problem. In 2013, according to Statistics Canada, about 20 per cent of all households (and about 60 per cent of households with members under age 35) used cellphones only. We no longer have the advantage of universal coverage from a single source. Some pollsters have responded to this by sampling among cellphone users in addition to landline sampling (called "dual frame" sampling), but this practice can add significantly to the cost of surveying and certainly does not enhance response rates, since few cellphone users will pick up their phones for pollsters.
The rise of interactive voice response (IVR) and Internet technologies are a mixed blessing for the industry. They offer significant relief from the rising costs associated with traditional interviewing and adding cellphones.
IVR involves telephone interviews conducted by computer along with random-digit dialling. While costs are dramatically lower, response rates are absurdly low, in the 2-per-cent to 3-per-cent range. Inexpensive Internet surveys typically use panels of respondents assembled through cold-calling, banner ads, list purchases from multiple providers and so on – methods that bear no relationship to random sampling.
With issues such as these, it's a wonder that any opinion polls today can accurately gauge the electorate or predict outcomes. But the changes are long-term and structural and are not going away. We have to live with it.