As the size and sophistication of these algorithms grow - as the number of variables they scan rises from dozens to hundreds to thousands - the confidence in their predictive power increases. Business writer Stephen Baker, author of The Numerati, has concluded that "the mathematical modelling of humanity promises to be one of the great undertakings of the 21st century."
Who governs the decision engines?
The algorithmic expansion may be great in scale, but is it also great for us? Clay Shirky, who has written extensively about the impact of technology, says from his office at New York University that to a large degree authority is shifting away from "experts" to sources of information that no one is really in charge of. "The workings are hidden and, in some cases, not understandable."
That means that the keepers of the algorithms have become figures of enormous importance. "In a society with an increasing amount of 'algorithmic authority,' " Mr. Shirky argues, "understanding who manages the algorithm that society relies on, and how those algorithms are shaped and used, becomes a key question of governance - just as, in a previous age, understanding the limits of military or police power became a key part of democratic governments."
The key issue, he says, is not whether algorithms have too much power, but whether society is able to keep them in check. "You would start worrying about a tool when it became powerful enough that the owners of that algorithm could start to shape outcomes to their own needs."
That is precisely the reason why many people are worried about the impact that algorithms now have on financial markets. To find algorithmic authority in full effect, you need look no further than banks, trading floors, investment houses and hedge funds.
Algorithms are now responsible for about 70 per cent of all trades in U.S. equities. They execute trades at speeds and volumes that were, until recently, simply unimaginable. The average number of shares traded on the New York Stock Exchange increased by 181 per cent from 2005 to 2009 and the time required to execute a trade dropped to 650 microseconds. Algorithms scan the markets for the tiniest imperfections and pounce with lightning speed, often holding shares for less than a second.
At those volumes and speeds, human intervention is simply not possible, and even the smallest errors can quickly spiral out of control. On May 6 at 2:42 p.m., the New York Stock Exchange dropped an extraordinary 600 points in just five minutes. Twenty minutes later, it had made almost a full recovery. Subsequent investigations revealed that the "Flash Crash" of 2010 was caused by a rogue algorithm unleashed by a mutual fund in Kansas City.
High-frequency trading has become a very profitable business on Wall Street. The handful of specialized firms that do it can bring in up to $100,000 a day. And the programmers who develop the algorithms that drive the trades have become the new masters of the universe.
The problem is that even the developers sometimes lose control of their creations. Earlier this year, Credit Suisse was fined $150,000 by the New York Stock Exchange for "failing to adequately supervise the development, deployment, and operation of a proprietary algorithm." It turns out that the bank wasn't aware that their algorithm was issuing thousands of cancellations orders for stocks that had not even been ordered.
In June, Delaware Senator Ted Kaufman warned that "our financial markets should not be reduced to a battle of algorithms in which capital formation is an afterthought, and long-term investors are relegated to second-tier status."
Are you a gadget? Please check yes or no
Mr. Kaufman's warning may have come too late. As algorithmic authority grows, we become more confident in the wisdom offered up by computer programs and less confident in our own decision- making abilities. We look to eliminate the guesswork and go for the sure thing, even in the creative arts.
A British company called Epagogix has developed an algorithm that challenges screenwriter William Goldman's famous assertion that, when it comes to knowing how to make a successful movie, "nobody knows anything."