Microsoft's most important new project in 25 years started out as an unmitigated disaster.
It was 2007. A group of executives gathered in a boardroom on the company's Redmond campus to watch a rough concept video of how Microsoft was going to regain its mojo in video gaming. With the company's crash-prone Xbox 360 platform starting to show its age, a team proposed to leapfrog Nintendo's motion-controlled Wii system—players would go controller-free, using only their body motions and voices to interact with the console. The man with the highest stake in the session was the newly appointed president of interactive entertainment—Don Mattrick, a Canadian video game wunderkind now facing the biggest test of his career.
To convey the concept, the team working on Project Natal had split the screen. Bad sign. Humans live in one reality, not two. On one side of the screen played footage from a traditional car-racing game. On the other side was an actor pretending to turn a virtual steering wheel. Except he looked like he was trying to open a bank vault while having a seizure.
The presentation went over like a lead balloon—and Mattrick realized he and the team working on what came to be called Kinect had to start over.
"That's what happens when you take a bunch of core game designers and just sit in a room coming up with ideas that you think are going to be broadening, with little to no understanding of your customer," Kinect creative director Kudo Tsunoda said this summer, looking back on the false start. Instead of just being video game designers, the Kinect team had to start thinking like regular human beings.
It was by no means certain that Microsoft could travel from that pivotal meeting to Kinect's current status as the fastest-selling electronic device in the world. The company's genes didn't augur well. As a popular org-chart spoof has it on the Web, Microsoft can resemble a set of distinct fortresses, each one poised to take shots at the others.
Yet Microsoft actually was ahead of most of its competitors in seeing the potential of video games to grow up, and even define the future of computing. But the company couldn't capitalize on the insight.
The original Xbox console, launched a decade ago, was designed to entrench Microsoft in the living room. The idea was ambitious but simple: Your Xbox would be the one device you turned to for entertainment, whether you wanted to stream a movie from your home computer or play a video game. That goal was so alluring that Microsoft didn't mind losing money every time it sold an Xbox, just so long as it got in the living-room door.
But the strategy wasn't working. For one thing, new offerings from Sony and Nintendo (the PS3 and Wii consoles, respectively) were eating up growing chunks of the market.
There were also technical glitches. By 2007, the Xbox was best known to many consumers not as the Swiss Army knife of digital entertainment, but as a technical train wreck. Users frequently complained of the "red ring of death"—a trio of lights that, once it appeared around the console's power button, indicated the Xbox's insides had pretty much exploded, and the console was now, in electronics parlance, "bricked." Microsoft eventually took a $1-billion charge on warranty extensions and replacements, to say nothing of the public-relations cost and the eventual fix (all currency in U.S. dollars).
To complicate matters, the video game industry is a fast and fickle one, where success can be as fleeting as a newbie's first play. Consider a recent example: Guitar Hero, the hit game in which participants "play" on a guitar-shaped controller. In 2007, Activision sold 1.5 million units of the then-current Guitar Hero title in its first month on the U.S. market. In 2010, that number dropped to less than 100,000 for a newer iteration. This year, Activision put the title on hiatus.
The same pattern holds for consoles. Even the Wii, the bestselling home gaming console of its generation, has seen waning sales for the past few years, forcing Nintendo this year to announce the development of a brand-new console.
But as that initial generation of motion-controlled games began to take hold in the mid-2000s, Microsoft executives knew that if they were to gain supremacy in the gaming market, they would need something future-proof, something that would buy the Xbox a few more years of good sales before Microsoft, too, would be forced to spend billions on a brand-new console.
When Microsoft came calling, Don Mattrick was already at the top of the video game heap. And he was bored.
There's an intensity about Mattrick, although it's difficult to see at first. He's a soft-spoken 47-year-old with the harmless good looks of a sitcom dad. But behind the calm, friendly demeanour he conveys in public appearances and interviews is an endless supply of weapons-grade ambition.
He's always been that way. As an adolescent, he walked into a ComputerLand store in his hometown of Burnaby, B.C., and demanded a summer job. There were no jobs available. So he began to work for free. Two weeks later, he was hired.
"He could sell a refrigerator to an Eskimo," the store's former manager, Wayne Galaugher, once told The Globe and Mail. "People trusted him on sight."
In 1982, Mattrick decided nobody was making interesting games for the PC, and started his own video games company with his friend Jeff Sember. Mattrick, 17, based Distinctive Software Inc. in his parents' home. The duo's first game, Evolution, had players working their way up the chain of life, from a single cell to a fully formed human. The feat won the young duo a spot on Front Page Challenge.
Mattrick was on his way. In 1991, at age 27, he sold Distinctive to California-based Electronic Arts for about $10 million, mainly in stock of the new company. He joined Electronic Arts and, over the next 15 years, helped turn it from a game-development shop with a market capitalization of $185 million into a $15-billion behemoth. He ran offices in the U.S., Canada and the United Kingdom, eventually heading up EA's studios worldwide.
Over the course of this work, Mattrick's critical talent—the ability to put together teams with various skill sets—emerged. It helped him produce some of EA's bestselling titles, including numerous sports-related games, such as the Madden NFL series, that offer unusually long-term profitability because they have a built-in fan base and can be resold every year with new team rosters and players.
More importantly, Mattrick was one of the first people in the industry to envision the socialization of gaming. One of the titles he helped manage, Ultima Online, was among the first online interactive games, a precursor to blockbuster multiplayer games such as World of Warcraft that today are among the most profitable titles in the business.
Team-building skills notwithstanding, perhaps Mattrick's biggest asset for EA was his ability to close the deal. Most lucratively, in 2000 Mattrick led the effort to buy the worldwide video game rights for the Harry Potter franchise. His pitch, which included an expansive digital universe for the story's characters and the potential to include concepts that players wouldn't find in the original books, won over author J.K. Rowling and Warner Bros., and EA won the deal. By one estimate, the nine Potter games have together sold almost 40 million units.
By 2006, Mattrick was a legend in the gaming industry. He had been named Ernst & Young's Technology Entrepreneur of the Year in 1998. Simon Fraser University gave him an honorary doctorate. "I loved the competitive side," he says. "I had a wonderful time at EA, but after 20 years in the gaming industry, I decided to take a year off."
During that respite, Microsoft's head of the entertainment and devices division, Robbie Bach, came calling.
Mattrick's first task as head of the interactive entertainment division at his new employer was to change the balkanized company culture. He immediately began trimming the fat, and focusing his staff on a few core projects.
The seed for one of the most important projects was planted years earlier, when Mattrick had a chance conversation with a film executive. The executive told him the single biggest hurdle keeping video games from expanding in the mainstream entertainment market was controllers—they were simply too confusing for the non-gamer. By 2006, Mattrick had seen the proof. That year, Nintendo unleashed the Wii.
At first, many in the industry laughed the Wii off. Ever since the dawn of the industry in the early '70s, video games had been a testosterone-laden domain, financed by the disposable income of males aged between 12 and 35. By the early 2000s, the industry was saturated with toys for the boys: Waste that alien, steal that car, blow stuff up.
Now here was this bizarre gadget with a funny name that used minimalist bars for controllers. At a time when the other manufacturers were cramming as many buttons and directional sticks on their controllers as humanly possible, the Wii came with controllers that harkened back to the days of the original Nintendo Entertainment System and its press-A-to-jump-press-B-to-shoot input scheme. A lot of people thought the Wii was a gimmick, a joke.
That is, they thought so until videos started popping up on YouTube of grandparents happily swinging those throwback controllers through the air, mimicking the movement of a golf swing or a home run. Nintendo suddenly made gaming accessible to everyone from small children to senior citizens. Entire families lined up for hours to get their hands on the console.
This was not something Microsoft could afford to miss out on. What had begun four decades ago as an arcade-based amusement for teenagers had grown into a $40-billion industry. Blockbuster game titles now rivalled Hollywood pictures for budgets and profits. And the content coming out of the video game world was continually expanding beyond the console and into the worlds of movies and other media.
Mattrick knew that if Microsoft was ever to fulfill the original vision for the Xbox—a central hub for all digital entertainment in the living room—it too needed to expand the console's appeal beyond hard-core gamers. He wanted to win over those families, the ones with the small kids and an insatiable appetite for every Disney cartoon ever made—the ones who would continue consuming all manner of media.
"Controller gaming is still super-important," says Mattrick. "But we want to redefine fully what an integrated entertainment experience will look like."
"We started thinking about what a family does in the living room," he says. "The living room is a big space, you have a big display, so you can take them to imaginary places."
And to get them, he needed to make a better Wii. So Mattrick instructed a team of developers to build him a completely controller-free system, one that functioned on movement and voice alone.
Mattrick assembled a grab-bag of skill sets: designers, programmers, mathematicians, folks who understood physics and anatomy. "We were looking to build a more visual interface," says Mattrick. "We wanted to explore natural interactions." For a company that has had the same dedicated silos for decades (see: Windows, Microsoft Office), it was a breath of fresh air.
The culture of the gaming division also had to change. "We had to pick our bets," Mattrick says. "There'd be less random experimentation and more high-performance teams. I'm a big believer in the wisdom of teams."
The technical hurdles of the new project were computationally crushing. Microsoft engineers estimated that a system for pure motion-capture gaming—the kind with no controllers—would need to calculate somewhere in the range of 1,000 variables in real time. Even using the most general assumptions about how the human body works, the combinations of possible hand movements numbered in the millions.
Microsoft began licensing computer vision technology from an Israeli company called PrimeSense, and bought another Israeli firm, 3DV Systems, for its 3-D-capture technology. Luckily, a number of employees at Microsoft's research lab had done postgraduate work on movement recognition, which the Project Natal team made extensive use of. Engineers began pulling voice-recognition code from the work Microsoft had already done for various carmakers' hands-free communication systems, such as Ford's Sync. Microsoft developers travelled the world collecting audio samples of various accents, dialects and languages to beef up the voice recognition.
After that failed 2007 presentation to Mattrick and other Microsoft executives, the Kinect team—which started with only about 15 people and eventually grew into the hundreds, including musicians, special-effects experts and artists—reworked the initial concept. They tried to get back to the basics of how human bodies work. If the Kinect was to one-up the Wii, it needed nuance.
They baked a high-end microphone and a depth-sensitive camera into the device, so it could recognize voice commands and multiple users at once. The team hired mimes to act out various motions. They studied Soundpainting, a gesture-based sign language created by New York conductor Walter Thompson for use in spontaneous composition of music and performance art. They hung out in various Seattle parks videotaping children at play. They adjusted the system to make it far more intuitive. The goal was to create a gaming system that needed no instruction manual.
A few months after the initial presentation, members of the team sat a nine-year-old named Noah in front of an early prototype. The game was a modified version of Burnout, a car-racing title. Within 30 seconds, Noah had figured out, unprompted, how to manipulate the game with his own movement. Within a few minutes, Noah's mother told him to take a break so she could play. The team knew they were on the right track.
The Kinect team's second video presentation to executives, although just as cheesy as the first one, hit all the right notes. Here were families in the living room playing various games by acting out the natural movements associated with those games in real life. And here was a different family challenging them online. The tag line was "You already know how to play."
Mattrick saw his vision becoming reality. He pushed the project forward, convincing Microsoft's top brass to tie up millions in investment dollars in the project.
The company launched the Kinect, named to reflect the ideas of movement and social connection, in November of 2010. For Mattrick, there was no need to be modest about what Microsoft had accomplished. The launch was a star-studded L.A. event, complete with a troupe of Cirque du Soleil performers to emphasize the theme of human movement. A voice on the loudspeakers informed the audience that "history is about to be rewritten." The first titles were aimed squarely at the kind of people who loved Nintendo's Wii. Kinect Sports, a collection of games such as bowling, soccer and beach volleyball, was a direct response to Wii Sports—except without the need for controllers.
The response was unprecedented. In its first three months on the market, Kinect flew off the shelves at a rate of more than 130,000 a day. In early 2011, Guinness—the world-records people—came calling. It turned out the Kinect had sold faster than any other electronic device in history, including the iPhone. By the spring of this year, Microsoft had moved more than 10 million Kinect units. Suddenly, everyone—including other Microsoft divisions—wanted to get their hands on the gadget.
"It feels almost as if it was the coolest sci-fi movie in your living room," says Wil Mozell, a co-founder (with Mattrick) of Vancouver-based gaming studio BigPark, lately part of Microsoft. "Kinect has rejuvenated the life cycle not just for Xbox, but for the console business."
The change was evident when Mattrick took to the stage in L.A. this summer to deliver Microsoft's keynote at the E3 conference. Most of the presentation centred on Kinect. Traditional video game titles, live sports content on ESPN, Bing voice-based search—these days, everything Microsoft develops in entertainment seems to have some sort of Kinect component. "The success of Kinect has, frankly, led to teams outside the interactive entertainment business looking at the possibility of having Kinect be a part of their experience," says Mozell of BigPark. "It's a significant change."
Indeed, Kinect has sparked a renaissance at the company that was coming to be seen as the sluggish grandfather of the software industry. And just in time. The success of Apple's iPad and the tablet market is dramatically slowing traditional PC sales, and Windows-based smartphones are still a distant fourth behind the three major platforms.
Indeed, Kinect's ultimate impact on Microsoft, consumers and the tech industry may have little to do with video games. A startling range of applications for controller-free interaction is already being mooted, from pro sports to medicine (see "This is not a game," page 36). But most lucratively for Microsoft, consider all those people out there who are completely baffled by computers, but can now simply speak or gesture what they want to do and—Hey, presto!—the computer responds.
Thanks to the spectacular success of Apple's mobile devices and Nintendo's Wii, the prevailing trend in digital consumer goods over the next decade will be all about dumbing down. Microsoft Windows is an incredibly customizable piece of software—users can tinker with just about every part of it. But the same customizability that gives hackers and enthusiasts the ability to do interesting things is also what brings countless ordinary users to the tech help desk, after they inadvertently delete an important system file.
There's little such concern with something like an iPad. That device adheres to what Apple CEO Steve Jobs has called a "curated" experience. The user doesn't see or tinker with any of the gears under the hood. They simply touch the screen, and the magic happens. Hard-core tech-heads may despise such an approach, but ordinary users love it, and they're the much, much bigger market.
With Kinect, Microsoft has a chance to capture the dumbed-down future of computing. Now, all a user needs to know is how to move a finger, how to say "Kinect: Do this," "Kinect: Do that." It could one day fundamentally alter the way people communicate with all their digital devices, from desktop computers to robot vacuums.
Suddenly, in the here and now, the only thing users (and, in some ways, creators) have to worry about is the experience.
The process of creating a traditional video game is usually split 80-20, with the minority of the time spent considering what the game experience should look and feel like, and the vast majority of the time spent actually developing the code. With a Kinect game, it's the exact opposite. "Kinect made us think about what it's like to be human beings," says Joe Nickolls of Microsoft Game Studios. "And that makes us think about games differently."
This is not a game This year, Microsoft began releasing tools that allow developers to build non-gaming projects using the Kinect gaming hardware. The results to date point toward applications that could change the way people communicate with all their digital devices (not to mention one another)
SPORTS (FOR REAL) In the world of pro sports, the ability to analyze the movements of athletes has become extremely important. Most traditional methods involve plastering athletes with bulky sensors. Two students from Lewis & Clark college in Portland, Oregon, calibrated the Kinect hardware's camera and motion sensors to develop a motion-based game of Pong. This prompted the idea that they could record and study the movements of the school's quarterbacks, for example, without having to outfit them with gadgets that hamper those very movements. If it works, Kinect-based athlete analysis could revolutionize sports.
DO AS I GESTURE At a Microsoft-sponsored developer camp held this summer, a group of programmers caused a stir by hacking a quadcopter—a nifty-looking remote-control helicopter. Normally, users control such devices with proprietary remotes or, in more recent cases, software that runs on their smartphones. But by using Kinect, the programmers built a hands-free control system—simply swing your hands one way or another, and the quadcopter flies in that direction. Such hands-free control could become the norm for a variety of applications, from the front door to the robot vacuum.
SMOOTHER SURGERY This March, a team of doctors at Toronto's Sunnybrook Hospital took the Kinect into the operating room. They set up the device to monitor the movements of a surgeon so that the doctor could control a nearby screen with nothing but hand gestures. The Kinect allows doctors to view reference images of the patient hands-free while working, so sterility is not compromised. While remote surgery using a physical controller is already a reality, Kinect may hold the promise of remote surgery that relies entirely on the movement of a surgeon's hands.
VIDEOS IN 3-D A researcher named Oliver Kreylos has been working on using the Kinect to capture and play 3-D video. Although the work is still in the very early stages, he has managed to produce YouTube videos in which he uses the Kinect's depth-capturing ability to give the images in the video an extra dimension. This allows him to move the camera around in real time, showing the depth of the images in the video. 3-D video-capturing could give audiences a whole new way to interact with the content they view.