Go to the Globe and Mail homepage

Jump to main navigationJump to main content

(Alexander Raths/Getty Images/iStockphoto)
(Alexander Raths/Getty Images/iStockphoto)

Data

How to spot bogus research Add to ...

The words “clinically proven,” loved and abused by infomercials and health-store potions, no longer carry the same weight they once did.

Still, the debate sparked by revised breast-cancer-screening guidelines last month raised questions for those who place their faith in well-designed, carefully executed clinical trials as the pinnacle of evidence-based medicine. Because it wasn’t just a clash of emotion with science: Both sides of the debate had a pile of studies and review papers from prestigious journals that appeared to support their positions.

How can two studies of the same topic reach opposite conclusions? Surprisingly easily, unfortunately. Stanford University epidemiologist John Ioannidis has famously estimated that 90 per cent of published medical research is wrong, thanks to factors such as sloppy statistics, inadequate study size and duration, and bias – both conscious and unconscious.

In the case of breast-cancer screening, for example, much of the data supporting routine mammograms for women in their 40s comes from observational studies, rather than more rigorous randomized trials, according to the authors of the new guidelines. On the other hand, critics of the new guidelines argue that they rely too heavily on older studies that used now-obsolete screening equipment.

The “right” answer is seldom cut and dried (except in the case of infomercials, which you can safely ignore). To reach your own conclusions about how reliable new health findings are, watch for these three common research failings.

CHERRY-PICKING DATA

The study

Last year, the Canadian Medical Association Journal published a review article showing that cigarette smoking can help marathoners run faster. The evidence was persuasive: Numerous studies show that smoking boosts lung volume and hemoglobin, and stimulates weight loss – all factors known to improve running performance.

The catch

Of course, the reality is quite different. University of Calgary medical resident Ken Myers wrote the article as a spoof, to show how easy it is to support virtually any hypothesis by carefully selecting your data. Whatever you’re trying to prove, “there’s always evidence if you look hard enough,” Dr. Myers says. “The challenge is figuring out whether that evidence really applies to the situation you’re interested in.”

For the record, the increased lung volume and hemoglobin in smokers are signs of respiratory problems – unlike in runners, where they signal adaptation to training.

Your defence

“Think critically,” Dr. Myers says. This includes considering alternate interpretations of the data, and asking who funded the study and who stands to benefit from its conclusions. Independent reviews from groups like the Cochrane Collaboration have strict guidelines for which studies can (and must) be included in any literature review, rather than leaving it to the discretion of the reviewer.

CORRELATION VERSUS CAUSATION

The study

In 2009, a study in the Archives of Internal Medicine made headlines with the finding, based on a 10-year study of 500,000 people, that eating more meat increases your risk of death. The results were “adjusted” to take into account confounding factors such as age, education, weight, exercise habits and so on.

The catch

This process of “statistical adjustment” is never perfect, cautions Stanford University clinical statistics expert Kristin Sainani. If multiple risk factors tend to cluster together in the same people, it’s hard to distinguish between them; and there may be other underlying risk factors that the researchers don’t even consider. “There’s always some residual error,” Dr. Sainani says, “and nutrition studies in particular are terrible for this.”

In the meat study, a closer look at the data reveals that eating more red meat also seemingly raises your risk of accidental death from car crashes and guns – a clear sign that the statistical adjustment failed to find all the underlying risk factors that affect meat-eaters.

Your defence

Poorly designed studies that fail to consider all major confounding factors can easily produce bogus risk increases of up to 60 per cent, according to Dr. Sainani. If a study finds that a food or medical intervention changes risk by more than 60 per cent, you can be fairly confident that the effect is real; for smaller effects, be skeptical.

FISHING EXPEDITIONS

The study

A 2008 Harvard study of 38,432 women looked for links between caffeine and breast cancer. They found three different patterns that were “statistically significant,” meaning there was less than a 1-in-20 chance that the apparent pattern was just a random fluke. For example, caffeine intake was linked to “increased risk of estrogen/progesterone-negative tumours and tumours larger than two centimetres.”

The catch

The researchers had analyzed 50 different possible caffeine-cancer links. Since you expect that one out of every 20 tests will randomly produce a false positive, the researchers should have expected about 2.5 of these false alarms – which is pretty much what they saw. As a result, Dr. Sainani notes, the results are most likely due to chance rather than any real link between caffeine and breast cancer.

Your defence

These sorts of “fishing expeditions,” in which large pools of data are systematically searched for any possible patterns, can serve a useful purpose, pointing out previously unsuspected links. But you have to treat them with great caution – at best as an indication that further research might be merited. Otherwise, it’s best for you to rely on studies which are specifically designed to answer a single question or a small number of questions.



Alex Hutchinson blogs about research on exercise at sweatscience.com. His new book – Which Comes First, Cardio or Weights? – is now available.

Follow on Twitter: @sweatscience

In the know

Most popular video »

Highlights

More from The Globe and Mail

Most Popular Stories