Just about every week, some researcher reports finding another statistical quirk in the equity market that might be harvested for a trading edge. Now a new paper says most of them don’t work as billed.
Looking at 447 supposedly repeating price patterns identified in the last few decades, academics from Ohio State and the University of Cincinnati contend that more than half are basically figments of their discoverers’ imagination. The study, “Replicating Anomalies” by Kewei Hou, Chen Xue and Lu Zhang, attributed the findings to a statistical sleight of hand known as p-hacking.
While lodged squarely in the academic realm, the paper is a broadside against an area of research that has come to dominate financial economics and underpin both quantitative investing and smart beta exchange-traded funds. It joins a growing body of literature that suggests people looking for money-making opportunities within the market’s chaos often see what they want to see, or confuse profitability with luck.
“Armies of academics and practitioners engage in searching for anomalies, and the anomalies literature is most likely one of the biggest areas in finance and accounting,” the authors write. “As hundreds of anomalies have been documented in recent decades, the concern over data mining has become especially acute.”
Nothing But Noise
Predictability in asset markets is a goldmine for anyone with the wherewithal to exploit it. A huge body of literature has blown up since the 1960s purporting to show that, from time to time, one thing leads to another in asset prices. A trader armed with evidence that a company’s profits or size holds clues to its future returns can mine that edge into perpetuity, and some of the great fortunes of investing have been built on such observations.
The authors’ inquiry involved taking market anomalies previously observed by other researchers -- say, that certain cheap stocks tend to move in a predictable direction -- then trying to replicate them in their own data. Often, they found nothing but noise. Of 447 anomalies examined, 286 generated statistically insignificant predictions for stocks in the category, and the record was much worse for certain kinds of trading signals.
Broadly, the authors said that the signals failed because their discovers had considered too broad a universe of stocks when they set out to confirm their findings. Tiny companies make up the majority of stocks, they noted, but represent very little market capitalization -- and yet that’s where a lot of the anomalies work best. They posited various flaws of analysis that led earlier academics to give too much emphasis to these stocks in their research.
“Why does our replication differ so much from original studies? The key word is microcaps,” the authors wrote. “Unfortunately, because of high costs in trading these stocks, anomalies in microcaps are more apparent than real.”
The researchers looked at a variety of factors, including momentum, value-versus-growth and ones based on trading frictions. According to their findings, nearly two-thirds of the market variations couldn’t be replicated 95 per cent of the time. Even for significant anomalies -- such as price momentum and operating accruals -- the magnitudes often are are far lower than reported. In other words, “the capital markets are more efficient than previously reported,” they write.
The authors said that identifying market anomalies is a “prime target for p-hacking,” which, generally speaking, refers to shaping data to fit a preset conclusion. The data are inherently empirical, giving researchers flexibility in manipulating samples, defining variables and choosing methods. There’s also a financial motivation for finding new aberrations “with trillions of dollars invested in anomalies-based strategies in the U.S. market alone.” And the more eye-catching the findings, the more likely the researchers are to get noticed and advance their careers.
The study concludes that researchers looking at anomalies should more closely connect their empirical work to existing economic theory. This would lessen the impact of data mining errors on their findings. And everyone involved in the research should be vigilant about the potential distortions from financial conflicts and publication biases.
“Authors, referees, and editors should be keenly aware of the complex agency problems that can arise,” they write. “Referees can be more open to papers that take care in developing well grounded economic hypotheses, even though their empirical findings might not be (that) significant. Without such a publication bias, authors would most likely have fewer incentives to engage in p-hacking.”