This is going to be a bit of a weird post, and also much longer than usual. But I wanted to take the time to show how difficult analyzing trends and interpreting primary data can be. Hopefully, it will educate readers and make you appreciate Deloitte research like TMT Predictions even more than you already do!
Last week, Nielsen released its quarterly Cross Platform Report. There were a few hundred articles written about the report in the days following, many of them referencing this column from Peter Schwartz in noted tech blog All Things D. He drew a bunch of conclusions, some of which I think may or may not be right, but were not justified by the Nielsen report.
Why does this matter?
- The current analytical consensus believes that online and streaming video is about to displace traditional TV.
- That belief is being reinforced by data such as the Nielsen report… but that data is (by and large) not being properly examined.
- People are making decisions based on that data. If the data is being wrongly interpreted, then people may make wrong decisions.
So think of this post as a primer on how to interpret data. I happen to use the Nielsen report as a case study, but the methodology will work for any media report.
Before we move on, two comments about Nielsen data. One, his data is not self-reported. These people are not being asked to remember what they watched or asked what they think they watch. Nielsen uses highly accurate passive measurement technology that tracks real-world behaviour. And two, the sample size is huge. Many studies look at a few hundred users over a few days and their statistical significance can be questionable. The trends discerned from tens of thousands of viewers over six months will be very valid. In other words, this is pretty much the most accurate consumer data you're going to see.
Are you ready? Is the PDF downloaded? Are your pencils sharpened? Good… because there will be a short quiz after.
1. Skip the introductory stuff, and start at the bottom of page two, in the box titled "methodology."
2. The breakdown of the 13,128 content streamers into five quintiles is perfectly acceptable. But, before analyzing any conclusions, it is really important to note that sixth category of non-content streamers - another 7,253 subjects. Out of a total of 20,381 subjects, Nielsen's charts that follow exclude almost 36 per cent of all those measured. That is a legitimate data analysis decision to make, but it is a huge number and the data that Nielsen shows later apply only to a subset of the total viewing audience.
3. They also did a quintile split on TV viewers. As you can see, there is a sixth category for those who don't view TV… and it has only 95 subjects in it. Roughly half a per cent (0.466 per cent, to be exact) of Americans studied don't view any TV.
That is a pretty stunning number. Interestingly, I don't think that got mentioned in any of the stories. Two reasons: I suspect most reporters and bloggers don't bother reading the methodology sections; and everybody knows that North Americans watch a lot of TV. Writing a story about "how TV is doomed" is fun and exciting and draws lots of hits and links. Writing a story about how TV is still being watched is less sexy.
4. In bold at the bottom of the page, the report authors say that their data reveals "two interesting and unprecedented correlations between content streaming and TV viewing." As always, please exercise caution when you see that word: as any stats person should tell you, correlation does not imply causation. The fact that a relationship exists is interesting and should be explored further, but is not meaningful on its own. Check out this page for some pretty humorous examples of things that are correlated, but almost certainly not linked in any causal fashion.