“We haven’t broken the Internet…” says Chris Jaffe, vice-president of product innovation at Netflix. He pauses a few seconds for dramatic effect before punctuating the pronouncement: “…yet.” Laughs break out in the streaming video company’s busy control centre, dubbed – appropriately enough – the war room.
Season two of the Emmy award-winning political drama House of Cards has just gone live and data has started to stream in from around the world. The launch is no small feat, considering that it goes live at the same time in each of the streaming service’s 41 countries. Netflix is now responsible for nearly a third of all downstream internet traffic, according to Waterloo, Ont.-based monitoring firm Sandvine. New episodes of its shows inevitably spike those already massive numbers.
A dozen employees are clustered around a long table, noses buried in their laptops, while a similar number are stationed at televisions around the room. Each person is tasked with making sure their individual piece of the larger whole is working properly. A gaggle of journalists look on and try to stay out of the way.
The room is plastered with House of Cards paraphernalia. A large upside-down U.S. flag – the show’s logo – adorns one wall, while a giant screen showing the Netflix home page and a live stream of fans tweeting takes up another. The menacing glare of Frank Underwood, the scheming congressman played by Kevin Spacey, is difficult to escape thanks to the ubiquitous posters. A lone vase of wilting white tulips sits in the middle of the table. The techies, busily typing into their computers, are oblivious to it.
Jaffe takes a roll call of the various platforms that Netflix’s 44 million subscribers use. “iPad? Check. Android? Check. Website? Check.” Somebody reports that season two has indeed gone live on the Nintendo Wii console. Another confirms that they’re seeing it on their smartphone.
Next up are countries. Brazil looks good, so does Mexico. Europe is okay. The 10 TVs around the room have the show running in 10 different languages. The French subtitles for Quebec are appearing without any problems.
It takes about a minute for the first episode of the new season to become the top-viewed show on Netflix in the United States, and about 14 minutes in Europe. It’s late night in North America but morning on the other side of the Atlantic – Jaffe explains that Europeans are probably more in the mood for cartoons right now than heavy political intrigue, but that will change in the evening.
The good news prompts a round of cheers. Champagne follows and everyone breathes a sigh of relief. A major launch has gone smoothly.
Netflix has rollouts like this down pat by now, having done it several times over the past year with such original series’ as Orange is the New Black and the fourth season of Arrested Development. The Los Gatos-based company is now comfortable inviting the press to witness its inner workings, a rarity for otherwise secretive Silicon Valley technology firms.
At the core these rollouts is a highly automated and decentralized system, where if one part breaks down, other backups and redundancies can step in to make sure things keep working. In that way Netflix is built very much like the Internet, a series of widely distributed nodes that the military originally designed to withstand nuclear attacks. The strain on the network imposed by a new season of a show such as House of Cards is therefore not unlike a nuclear strike on the Internet itself.
The company is a veritable poster child for cloud computing. Rather than spend resources on huge data centres in each country, it instead relies on Amazon to do so. Netflix then stores its TV shows and movies with Amazon Web Services, the book seller’s cloud hosting division, then supplements that with smaller servers within each country that local internet providers can connect to. In Canada, both Bell and Telus are part of this content delivery network.
This distribution across services lowers costs for everybody, from Netflix down to its subscribers, while maintaining a reliable, fast and dynamic system.
“The goal for us is to build a system that automatically adjusts our capacity so we can grow and shrink as customer demand ebbs and flows. It shouldn’t matter when the biggest spike will be because we’ll be ready for it,” says Yury Izrailevsky, vice-president of cloud computing and platform engineering. “It’s really the elasticity of the cloud being able to spin up additional capacity within seconds that’s allowing us to do this.”
While the war room is currently abuzz, the rest of Netflix’s headquarters is a figurative ghost town. A large theatre near the entrance of the main building, not unlike the kind you’d find at a multiplex, sits empty. A sign out front lists screening times for the first episode of the new House of Cards, which employees were able to take in a day before the official launch. Meeting rooms named after movies – there’s one for Batman, another for Lord of the Rings – sit unused, but that will probably change in the morning.
Despite housing some 1,200 employees across five buildings – with more being built – the company can effectively run on a skeleton crew, Izrailevsky says. Its systems are largely automated and support staff are only called when problems are detected. And if they do get roused in the middle of the night, they can usually fix most issues from their laptops at home.
It’s not a perfect machine, though, as Netflix does occasionally go down. The causes can vary, from malfunctions on the devices that viewers use to watch to problems with local Internet providers, but only rarely does something happen on the cloud side of things. “We aim for four nines – 99.99 per cent there,” Izrailevsky says.
If the champagne and cheers are any indication, the company has succeeded at that goal: a new season of House of Cards has arrived, and the Internet isn’t broken. At least for now it isn’t.