When pollsters get it wrong: the high cost of lost credibility

Share

Getting it wrong on the B.C. election and in other political races is not simply an issue of wounded pride for many of the country's political pollsters; the latest electoral flub has sparked a new wariness among companies that pay some of the same firms to survey their customers.

The same methods that mistakenly predicted, with one exception, an NDP win in British Columbia's vote earlier this week are used by those same polling companies on behalf of corporate clients trying to measure market share and customer preferences, or to gauge public perceptions of industry.

While some companies believe their market research is sound, many in the polling industry say they are re-evaluating the way they do things in the wake of botched political polls. Some suggest that market research – the work on which pollsters make most of their money – will also find itself subjected to much more scrutiny.

Wynne Powell, chief executive officer of London Drugs Ltd. in Richmond, B.C., said he has been wary of polling results over the past couple of years because the surveys are often conducted over the Internet, where, he said, respondents face no pressure to reply honestly.

"When you sat around the executive table and discussed polling results, you used to be able to really take those results pretty well to the bank," he said. "I've been more cautious to react to polling provided to me, especially if it's done on the Internet."

He said he's warned his staff to be "very cautious before we make any broad strategic changes in direction based on polls." And his firm now pays 20 to 30 per cent more for more extensive market research that includes Internet polls, phone polls and in-depth exit interviews of shoppers leaving his stores.

As a result of election-prediction misfires both in B.C. and last year's Alberta election, the Canadian Association of Petroleum Producers – which commissions polls in major cities outside of Alberta to assess the attitudes of Canadians regarding the oil industry – now looks for polling firms that focus on momentum, and where public opinion is headed.

"The perception that the pollsters got it wrong impacts their overall credibility amongst business leaders," said Janet Annesley, communications vice president for CAPP. "The oil and gas industry is driven by technical people. We are data-driven and risk-adverse," she said, adding the engineers, geologists and accountants in the oil patch want to know what exactly in the social science process needs to be fixed.

Jack Bensimon, president of Toronto advertising agency Bensimon Byrne, said the failure of the pollsters in B.C. will undoubtedly increase scrutiny of polling methodology, especially in light of problems that have already emerged in getting certain people – such as younger Canadians – to answer polls, and the fact that more people are disconnecting their home phones.

"There are growing concerns among marketers and advertising agencies, about the reliability of the data," he said. Pollsters are going to have to put more effort into weighting responses for underrepresented groups, he said, and they may have to use much larger samples to increase accuracy.

Mr. Bensimon also said that questions about polling accuracy were percolating long before the B.C. election. Inaccurate polls failed to predict the outcomes of the last Alberta provincial election, he noted. "I don't think that simply because the pollsters got it wrong, yet again, that market research has suddenly been invalidated," he said, adding that political pollsters will now seek to fine-tune their methods, with the benefits filtering back to market research.

Lorne Bozinoff, the founder of Forum Research Inc., which was the only polling firm to come close to the right call before the B.C. vote by predicting a slim Liberal majority (although that poll still gave the NDP a slight lead), said calling the outcome of elections in his business is mostly an "ego thing among pollsters."

But getting it right on election night does help firms market to corporate research clients. And it is the only way, Mr. Bozinoff says, to find out how accurate a pollster is, since with most market research, results are never subjected to the trial-by-fire of an election.

"Our clients like it when we call an election. They liked our results in B.C.," Mr. Bozinoff said. "It just gives them an assurance that the work that we do for them is probably also accurate."

Other retailers already questioned the value of market research, even before the B.C. election polling controversy. Christine Day, chief executive officer of Lululemon Athletica Inc., said in a recent interview that market research is often of limited use by the time it reaches the client. "I'm far more interested in what could be – what's needed – than I am in what is, because by the time there's evidence somebody else is already doing it."

Yet other executives say the recent political miscues have not undermined their faith in market research. Susan Senecal, chief marketing officer for A&W Food Services of Canada Inc., said the hamburger chain uses a wide array of methods for getting to potential consumers, including telephone and Internet surveys conducted by market research firms. Most of the questions are lifestyle questions, including what time of day people are the most busy or when they're most likely to eat out. This helps the chain make decisions when it comes to questions such as restaurant operating hours.

"There's a difference, I think, from politics, where you get just one decision point," Ms. Senecal said from A&W's Vancouver office. Through the restaurants themselves, "we have the advantage of having a whole gigantic pool of people every single day that we can go to to check our assumptions."