The election of Donald Trump to the American presidency was not supposed to happen. On Tuesday, virtually every poll in the country predicted his defeat. Poll aggregators gave Hillary Clinton an overwhelming likelihood of winning.
But again, as they have so often in recent years when the stakes were highest, the polls got it wrong.
That this failure happened on the industry's biggest stage, in the world's most consequential election, has brought renewed attention to defects and biases in the world of polling that have roiled 21st-century politics and mirrored some of the developments that pollsters have failed to predict.
"There's nobody draped in glory today," said Darrell Bricker, CEO of Ipsos Public Affairs, whose firm gave the former secretary of state a roughly 90-per-cent chance of winning on election eve.
Analysis: Woe to us who belittled the clout of the white working class
Opinion: What the U.S. pollsters missed
Read more: The surprising data behind Trump's win
In the final week of the campaign, most national surveys gave Ms. Clinton a four-point lead. On Wednesday, with late ballots being counted, she was expected to win the popular vote by just over 1 per cent, while losing the Electoral College that determines the presidency.
Even sophisticated data parsers like Nate Silver of FiveThirtyEight.com, which accounts for the quality of surveys and other factors in producing its polling averages, made Ms. Clinton the overwhelming favourite. His site gave her a 71.4-per-cent chance of winning on Tuesday, while the New York Times was even more bullish on a Clinton win.
That wide-ranging failure comes down to two factors, pollsters and political observers say – one of them inherent to polling and one foisted on the industry by technological change.
The first is that voters are often reticent or dishonest about their leanings. Henry Olsen, a senior fellow at the conservative Ethics and Public Policy Center in Washington, argued that pollsters overestimated the number of people who would vote for third-party candidates by taking those voters at their word.
On Monday, he suggested in an op-ed that many white conservatives nursing a sense of cultural grievance would come home to Mr. Trump in the final accounting and that national polls were lowballing his support. Sure enough, the Republican's lead among whites without a college degree spiked from 30 per cent in pre-election polls to 39 per cent in exit polls.
"Some of it is people lying to pollsters to save face – some of it is people really making up their minds on the way," he said in an interview on Wednesday.
The failure to discern these "shy" Trump voters is also a product of educated, affluent people – pollsters among them – becoming isolated in urban cocoons, Mr. Olsen argued.
"There is such a thing as an elite bubble that encompasses the centre left and the centre right. … They don't know people who think different than they do," he said. "I think a lot of pollsters try to fit people into the world they inhabit instead of trying to find out about the world that's out there."
Given the Brexit vote this summer and last year's surprise victory for the British Conservatives, which surveys failed to predict, news consumers should know that silent majorities often lurk outside the scope of modern polling, Mr. Olsen contended.
"The first time, shame on you," he said. "But the second, third, 18th time – shame on me."
Canadian polling veteran Angus Reid said the reason disaffected white voters are so elusive is that they are reluctant to take part in surveys. His firm showed Ms. Clinton with a four-point lead on Nov. 4, but he noted that even private polling for the campaigns missed the breadth of Mr. Trump's support.
"Older males just don't want to talk to pollsters," he said. "These aren't people who are in a big hurry to talk to what they see as the agents of the establishment."
It was a fitting challenge for a campaign largely defined by the alienation from mainstream America felt by less-educated white voters.
"It's people who I think are somewhat estranged from the polling industry," he said. "The industry has got a lot of issues that it's facing – and the main one is just who's willing to participate in surveys."
The problem can cut both ways – the difficulty of reaching young voters without landlines can also skew polling in favour of the old. And while most pollsters weight their results based on different demographic response rates, that process can distort their findings.
As Mr. Reid conceded: "This is an imprecise science."
Of course, some pollsters were more precise than others. TechnoMetrica Market Intelligence, a relatively minor player in the industry, produced the only national, four-way poll predicting Mr. Trump's victory. (It also wildly overstated the support of Libertarian candidate Gary Johnson.)
Raghavan Mayur, president of the firm, said that he succeeded where others failed by accurately gauging the higher level of enthusiasm among Republicans and right-wing independents. His pollsters asked prospective voters how "interested" they were in the election, on a scale of 1 to 7, with likely Trump voters consistently scoring higher. That result could help explain disastrously low turnout for Ms. Clinton in solidly Democratic states like Wisconsin, which Mr. Trump carried.
"My models worked," Mr. Mayur said.
And yet even as pollsters checked the box score and licked their wounds, some acknowledged the obvious fact that the malaise afflicting their industry pales next to the political turbulence ahead.
Mr. Reid, speaking from California, said he was reeling from the election of a president who, in Mr. Trump, "crossed all the lines of civility and decency."
"I think the emotional assault for me has been much more powerful than the intellectual assault," he said. "There are bigger issues than the future of polling."
Eric Andrew-Gee