Looking to chill a room of happy broadcast-industry folk? Bring up the subject of the portable people meter (PPM), a pager-sized device that picks up encoded signals from nearby TVs and radios.
Specifically, ask them how they feel about the sample sizes used to produce Canadian broadcast ratings. Then, stand back.
Recently, CFL commissioner Mark Cohon lamented the volatility of his league's ratings on TSN when so few people can sway the numbers.
"In Saskatchewan and Manitoba, the sample size is so small [fewer than 100 people]that people coming in and out of games has the effect of skewing numbers," he told Usual Suspects. "If just 10 or 12 people come out it can have a profound impact."
One TV broadcaster believes the system is better for sports, game shows and reality shows because, as opposed to older, more educated, more affluent news viewers, those audience types are willing to wear a PPM on their belt for three months. Another veteran of radio laments all it takes is one or two people in his demographic to change habits and he's a dead man in the ratings.
There's fear and loathing in the land – even among the people who benefit from the change to PPMs from a diary system. It's not unlike the criticism of political polls that sample 1,000 people to predict a national election result.
How can so few represent so much to so many?
To the untrained eye, sample sizes appear small. According to broadcast measurement and consumer behaviour data specialist BBM Canada, 9,000 randomly-chosen panelists are representative of the diverse tastes and habits of the Canadian population.
BBM says 90 per cent of that group wear their PPM on any one day, and they average 14 hours of listening/watching a day.
BBM's national PPM panel has "approximately 4,300 households across Canada. Market panels in Montreal, Toronto, Vancouver and Calgary report local audiences. Other homes located across Canada report national and regional audiences."
Regionally, that sample size breaks down as: Montreal (francophone) 500; Toronto 800; Calgary 529, Vancouver 672; national balance 1,804. Total: 4,305.
Still, just 800 people define the huge Toronto market?
"Sample sizes have always concerned people who aren't involved in the research," says Debra McLaughlin, CBC's senior director of research. "BBM has rigorous methods to test sample sizes. We're confident they're getting numbers right within the methodology. Week after week, the consistency of the numbers they come up with gives you a great deal of confidence about the numbers."
Broadcasters do sit on the BBM panels to assess the PPM methodology. When something goes awry from year to year – as happened when TSN's 2011 average-minute regular-season ratings for CFL dropped by almost 20 per cent – they can check back to see where and how the change was affected. Likewise they can mine the changes from diaries to PPMs for better demographic results.
"I know our general view of a hockey fan has changed as a result of PPM," McLaughlin says. "We can see we have more females tuning [in] More age groups tuning in. So in that sense, PPM has been great to unearth that secondary or tertiary audience."
It also means you don't overreact to changes between the old and new system.
"Every methodology has its benefits. With diaries, people routinely would write in, say, Seinfeld every night in that strip of programming. PPMs get the real listening or viewing pattern. These numbers should not be absolutes. If the No. 1 show in Canada goes down on PPMs it doesn't always mean fewer people watching. It just means there's a new way of measuring it."
Costas continued
Usual Suspects received some flak for a critique of Bob Costas's NBC interview with alleged pedophile Jerry Sandusky last Monday. Specifically, Costas was expecting to interview the former Penn State football assistant coach's lawyer only, and had just a half-hour to prepare when told Sandusky himself would talk via phone.
One reader said we needed to include the haste as a mitigating factor.
The surprise guest will make a fascinating anecdote in Costas's bio. But in journalism, we don't footnote interviews. Nor do we judge game reports done on tight deadlines differently than we do feature pieces that take months. Costas is paid big bucks – make that stupendous bucks – to operate on tight turnarounds.
We still thought it was a charged piece of theatre and have tremendous respect for Costas's body of work. His style of closed-ended questions and dramatic accusations is a staple of U.S. network journalism perfected by 60 Minutes. But we also believe he swung and missed when it came to finding out why Sandusky wanted to address the American public.