Liberal Leader Justin Trudeau, shown June 12, 2013.SEAN KILPATRICK/The Canadian Press
News stories about polls never get as much attention as when the survey results are unusual. Compared to a stable trendline, a big gain, a steep loss, or a wide lead is far more interesting, regardless of whether or not the result is accurate. In most cases, in fact, that extra attention is being given to what is known as an outlier. But can we identify an outlier when we see one?
A survey that was released last week looked like an example of such an outlier poll. Conducted by EKOS Research, the poll gave the Liberals 36 per cent support to just 26 per cent for the Conservatives.
There were a few reasons why this poll looked like an outlier. The Liberals had not been awarded a lead of this size in any poll since May, just after Justin Trudeau became leader. The Conservatives had also not been registered at this level of support since then, and this was only the second poll to put the Tories one point out of third place since the last election (and likely since the merger of the Canadian Alliance and Progressive Conservatives in 2003).
The results also went against recent trends, which had generally shown a small uptick for the Conservatives and a decline in Liberal fortunes since the spring.
Spotting an outlier
But an outlier poll is not necessarily the product of shoddy polling, as they can occur even when everything is done correctly. Polls often come with a margin of error of about three points, meaning that the reported support of a party can vary by roughly three points in either direction (the degree of variation is also partly dependent on the amount of support itself – the margin of error for a party at 50 per cent is not the same as one at 3 per cent, for example).
In addition, this margin of error is given with 95 per cent confidence, meaning that 19 times out of 20 the result will normally fall inside of the reported margin. An outlier poll, then, can sometimes be the victim of statistical probability.
(For more on how polls are conducted – and which methods are most reliable – read our methodology series.)
Some polls can be obvious outliers. One survey that was released in the last week of the federal campaign in 2011 gave the Conservatives 46 per cent of the vote, which put the party eight to 10 points above other estimates and 20 points ahead of the NDP. That would have given the Conservatives the best result in any federal election by any party in over 25 years. In the end, the polls that registered the Conservatives at 37 or 38 per cent of the vote in the final days of the campaign were closer to the mark: the Tories took 39.6 per cent.
On the other hand, what looks like an outlier poll can instead be the first indication of a new trend unfolding. EKOS was in the field Oct. 10-14, while the previous set of polls were out of the field by the middle of September. Perhaps there had indeed been a rebound of Liberal support in the interim.
The chart below shows an example of what seems to be, at first glance, an outlier poll.
The party represented by black dots has suddenly registered over 35 per cent support (highlighted in green), after rarely doing better than 25 per cent (and usually below that level of support) in earlier polling. The party represented by dark grey dots has been steadily leading the race, and by a wide margin, for some time.
While the black-dot party has been improving in the last few days, a jump to over 35 per cent and a lead – the first time any party other than the grey party has led in a poll – seems unreasonable. This is probably an outlier, but it would be unwise to pass judgment so quickly. Red flags should be raised, but we should also see what the future holds. Is this an outlier poll, or the start of a new trend?
Below you will find the same chart, extended into the future and with the parties represented by the dots identified.
That outlier poll (highlighted in green) was the first poll of the 2011 federal election campaign to show the New Democrats leading in Quebec. Conducted by CROP for La Presse, it was a surprising result that changed the perception of the campaign in the province. While some national polls were also hinting that the NDP was flirting with the lead in Quebec, this was the first survey with a very large sample conducted by a Quebec-based polling firm to show that the NDP was making a big breakthrough and that the Bloc Québécois was in serious trouble. What initially looked like an outlier turned out to be the first snapshot of the Orange Wave. Subsequent polling made this clear.
The same sort of approach needed to be taken with last week's EKOS poll, as it needs to be done with any other poll which appears to be an outlier. And with good reason: yesterday, Nanos Research released a new poll showing the Liberals enjoying an eight-point lead with 37 per cent support to 29 per cent for the Conservatives and 23 per cent for the New Democrats.
(Another poll released yesterday evening, by Abacus Data, showed a tie between the Liberals and Conservatives. This does make it more difficult to determine the actual state of the race, but this is consistent with the recent tendency of some polls showing a wide lead and others showing a tie. The truth likely lies somewhere in the middle.)
Considering the margins of error of the two polls, EKOS and Nanos have shown broadly similar results. The poll last week, the first to show that the Liberals may have regained the sort of lead they were enjoying before the summer, no longer appears to be an outlier. And with Conservative trouble in the Senate at a high simmer, it seems likely that their runner-up status could persist.
Éric Grenier writes about politics and polls at ThreeHundredEight.com.