How was a bug of the magnitude of Heartbleed introduced in the first place – and why did it take over two years to find?
It's easy to jump to nefarious conclusions – that intelligence agencies or ill-intentioned attackers surreptitiously slipped in the requisite code. But, the reality is, "writing security software, cryptographic libraries, and programs that secure Internet traffic, it's really difficult," said Seth Hardy, senior security researcher with the University of Toronto's Citizen Lab.
"A lot of people don't really understand the incredible amount of detail and attention to every possible outcome that needs to be made, because one mistake in the entire library can bring a system down. And that's a flaw of the type we're seeing with Heartbleed."
How do Internet security experts explain the recently discovered security nightmare? "It's like if your water company said 'All of our water is poisoned – and by the way, the only thing you can do is not drink water,' " says Matthew Green, an assistant research professor at the Johns Hopkins Information Security Institute specializing in cryptography.
"It's a pretty nasty kind of thing to hear. It's not something that's wrong in your house, it's not something you can even fix. It's something you just have to wait until the people who are in control of it get around to fixing it."
For most people, the Internet is Google, or Facebook, or the Yahoo homepage. It's an icon on a desktop, a menu bar or a dock. Software, services, protocols and servers – that's the invisible Internet: integral to making everything work, of course, but hard to fathom from within the confines of a browser.
The only glimpse of that side the general public gets is when something goes wrong.
The Heartbleed bug, made public Monday, is about as wrong as wrong gets. It turns out an important piece of software used to secure connections between users and websites was broken – and had been that way for two whole years. If left unpatched and exploited, the Heartbleed bug has the potential to expose usernames, passwords, and even cryptographic keys – the latter crucial to scrambling and descrambling all of the data a website sends or receives.
It may alarm some people that much of the Internet's most crucial software – the stuff that giants like Google, Facebook or Yahoo use – is actually developed by volunteers, non-profits and organizations kept alive on the goodwill of what small donations pour in each year. Companies big and small can use such software because it is freely available, or open source – and they aren't required to contribute changes or donations in return, although it is encouraged.
The OpenSSL Software Foundation, which funds the widely used software affected by the Heartbleed bug, "made less than $1-million last year, almost entirely in consulting contracts," according to The Wall Street Journal's Danny Yardon.
"$2,000 in outright donations, received in small increments mainly from overseas supporters of encryption, was not nearly enough to initiate a deeper revamping of the underlying code."
The project is managed by just four core European programmers, Yardon writes – and only one works on the project full time.
The thinking has long been that, since such open source projects as OpenSSL have hundreds or thousands of contributors over the project's lifetime, there are always eyes watching the code. But in practice, just because anyone can look deep into a piece of software's code, doesn't mean they will.
In fact, at many companies, it would be a full-time job. "But money and support still tend to flow to the newest and sexiest projects, while boring but essential elements like OpenSSL limp along as volunteer efforts," writes Rusty Foster in The New Yorker.
One thing that could make work on OpenSSL sexier are the reports of mass electronic surveillance by government intelligence agencies. As a result, more researchers are scouring the Internet's most crucial, underlying code – and, according to Green, it's one reason why bugs such as Heartbleed have been found.
"There's a huge new effort by researchers to look for exactly this kind of thing. So it seems like security getting worse, but security is actually getting better as a result," he explains.
And perhaps it's a good thing when a disaster like Heartbleed makes the public aware of the chaotic, hodgepodge underpinnings on which the entire Internet is built – largely on the backs of volunteer coders and developers who, given the importance of their work in our day to day lives, aren't paid nearly enough.
"It's a very easy way of thinking to say something like that is so major it must have been intentionally introduced, or must have been known about by those governments – like there's no way they could have missed this," says Hardy.
"But the reality of it is, it's easy to miss something like this, because doing this sort of work is incredibly hard."