Go to the Globe and Mail homepage

Jump to main navigationJump to main content

Duncan Stewart

For an analyst, the devil is in the data Add to ...

5. Check out the chart at the bottom of page 3. First, the nice smooth curves are drawn by the report authors. I think the curves are artistic…but equally I think they are potentially misleading or confusing. Straight lines wouldn't have looked as good but would do away with the suggestion that Nielsen has intermediate data points along the X axis, which they don't.

6. Yes, it seems clear that those who stream the most (quintile 1) watch the least TV. But the relationship is not linear or true across the quintiles: Those who stream the second most also were the second heaviest TV watchers in Q4 2010. The third heaviest streamers were the heaviest TV viewers in the same period.

For the Q1 2011 data, the data is equally variable: Second heaviest streamers were once again the second biggest TV viewers, for instance. But the datum that jumps out at me should be very worrying for those trying to draw conclusions: The fourth heaviest streamers (only streaming about 18 seconds of video per day) watched the fourth most TV and were not statistically significantly different than quintile 1 who were streaming almost 19 minutes per day (the difference between quintiles 1 and 4 was only 0.5 minutes, or 0.2 per cent).

If there is a relationship here, it is not linear. The statement, "the more you stream, the less TV you watch" is not supported by the data. You can accurately say, "the heaviest streamers watched less TV than the other 4 quintiles." Which is interesting indeed, but not what most articles said.

7. There is a real difference in the variability of the data on the two axes. The streaming axis is highly variable and skewed: The daily streaming in Q1 2011 range from 0.1 to 18.8, for a midpoint of 9.45 minutes plus or minus 99 per cent. Meanwhile, the TV viewing numbers range from 272.4 to 290 per day, or 281.2 minutes plus or minus 3.1 per cent. I have a major problem with drawing conclusions when one variable has such a broad range and one such a narrow range. Even very small errors could have large effects.

8. It is also worth noting that the quintiles for streaming content are badly clumped. The first quintile of heaviest users is nicely separated from the pack, but the remaining four are distinguished by only three minutes of streaming video per day. That's not a lot of variation to try to figure out patterns and quarter over quarter trends.

9. Another possibly confounding factor is seasonality. As Nielsen notes in every report, Americans TV watching is seasonally variable, which we can see by the TV viewing midpoint being higher in Q1 2011 than in Q4 2010, up about 5.4 per cent. One would think that streaming content and TV viewing would be similarly affected by seasonal factors, but no one have ever established that, to the best of my knowledge.

10. On to page 4, and the top left hand chart. Subset analysis is always fun - but because the number of subjects is lower (not stated in the report, but I would assume about 25 per cent of the subjects are in the 18-34 demographic) the data is also statistically less valid.

This one has many of the same problems as the charts on the preceding page: The second through fifth quintiles are tightly clumped with meaningless differences in streaming minutes; the TV viewing variability is wider (229.3 plus or minus 7.5 per cent, or double the range of all viewers) but that could be due to the smaller data set and is still quite a narrow range; and the second heaviest streamers are in a virtual dead heat to be the heaviest TV watchers. Nielsen is correct in saying that "more pronounced behaviours emerge," but I am not sure that isn't just an artifact of the subset size.

Single page

Follow on Twitter: @dunstewart

In the know

Most popular video »

Highlights

More from The Globe and Mail

Most Popular Stories