Skip to main content

The first part of my interview with Microsoft user interface guru Bill Buxton ended up being about the back stories of several modern consumer devices, from touch screen phones to smart watches.

The second part focuses on Kinect, the motion-based, controller-less interface that will come to the Xbox 360 this November. Mr. Buxton was only on the fringe of the device's development, but he's clearly excited about what it can do and where the technology will lead.

Did you have anything to do with the development of Kinect?

Only peripherally. When the team was brainstorming concepts at the beginning I would come in and help out. The credit goes to a group of people at our Microsoft research lab in Cambridge, England. They are really high-end computer vision people. They got the 3-D imaging working. There were also folks in the industrial design group. It was a really interesting, spread-out project. They would drop in people like me from time to time to just troubleshoot ideas and help do reality checks.

Has the technology behind Kinect been around for a while? Or is it something completely new that Microsoft Research dreamed up from scratch?

There's a really nice story around that. We can bring together everything we've been talking about in terms of Kinect.

For me the grandparent of Kinect is the door at the supermarket. If you think about it, it's got a one-bit computer, because a computer is just a bunch of switches. When you walk out of the supermarket with your arms full of groceries, the exit door opens automatically. It senses when you're coming. It basically has a proximity sensor. It only opens when you get close. There's an optical vision system, and it's hooked up to that door and it triggers a switch, which is the computer, and it opens the door. And I think you could argue that that is the grandparent of Kinect.

This technology started getting my attention in 1982. I used to work on video games for Mattel. They had something called Intellivision, which was an early game engine. It was the days of Coleco Vision and the first wave of game machines. I flew back to Toronto through Chicago on the redeye at around six in the morning and I can remember staggering into the washrooms at O'Hare Airport when I was changing flights. It was the first time I encountered a washroom where the taps went on and the toilets flushed and the hand blowers went on without ever touching them. I remember thinking: Wow, this is amazing. All of these appliances can sense my presence and what I'm doing and do the right thing without any physical contact. I said to myself these are just switches with sensors. Well, my computer has millions of switches, and none of those switches are as smart as the switches I saw in that washroom.

So that's the precedent.

I worked on the original Amiga 100. It was a great computer with great audio and graphics at the time. This was around 1986 or 1987. And this Toronto company called Very Vivid-they're now called GestureTek- built video games for the Amiga. I worked with them; I was consulting with Commodore at the time. They were working on technology where you could stand in front of games and do very primitive things with 2-D recognition. It was very simple stuff, but it was a precedent for Kinect, too.

Kinect is really neat because it's compact, it's got a 3-D camera, and the quality of the vision and the gesture recognition is just fantastic. It's an evolution.

But here's the kicker: If we had released Kinect 10 years ago it wouldn't have had anywhere near the same impact. It comes back to this larger ecosystem in gaming. There are other things that are really important here besides just the technology.

To start, gaming has moved out of the 12-year-old boy's bedroom-to use the stereotype-and into the living room, which is significant in a couple of ways. The boy's bedroom just had a little TV monitor whereas the living room suddenly started to have a 40- or 50-inch flat panel screen or projector system. Plus, game machines were generating fantastic graphics. Games are now doing things in real time that you needed a supercomputer to do when I started in the 3-D graphics industry for visual effects in Hollywood. The better graphics and bigger screen allowed games to start to become more cinematic, more engaging, and really exciting. Watching your kid play a game in cinematic form on this big screen was nearly as engaging as watching them play ice hockey or soccer at school.

Then at a certain point people like me-parents-start to say: How can I get involved? And to its credit, the Wii demonstrated that there could be a different kind of gaming for the rest of us: gaming for non-hardcore gamers. It's changed the nature of the living room, and it's changed the way gaming fits into the family structure.

Things like Kinect are just the start of this transformation of video games and how they fit in. You can't talk about something like Kinect in isolation as just the technology. You needed the games, you needed them to be in the living room, you needed the big screens to motivate the games' move to the living room, you needed the game content to be visually engaging so that other people in the family who weren't hardcore gamers would still be engaged in the activity.

Would you care to prophesize the future of motion-based technology?

I can see the game pad and other physical devices being used in conjunction with Kinect. And that opens the door for new types of games. I'll give you an example. You and I are hardcore gamers playing a traditional game, like hockey. Other people around us could be interacting not with controllers but with Kinect. They could be characters in the audience initiating a wave. Or, being a true Canadian, I could throw an octopus onto the ice to just screw up your game. You could be the crowd as opposed to the hockey team. There's a range of possibilities that I think game developers are going to start to see.

I also think the Kinect technology will move from the living room. We'll perfect it, get the economies of scale to push the price down and build up our experience with it. It comes back to what you said at the beginning of our talk, about creative uses of technology. My whole history has been making technologies that support people doing creative things like industrial design, filmmaking, 3-D animation, and especially music and audio. And what's fascinating is that everything from those "artsy-fartsy," not-so-serious, creative things-all of these technologies turn out to have a massive impact on mainstream applications.

To take Kinect, I can imagine that when you walk through Toronto the ads on bus shelters will be digital displays. You'll be able to interact with them via gestures as you walk toward them. I think that when you're on the subway and you see those big ads on the platform, they will be digital and detect your presence and you will be able to interact with them using something like Kinect. Like the switch at the supermarket opening the door, Kinect will be the technology that opens up the window for you to interact with ads. That sort of thing is pretty much inevitable.

So it will be another case of gamers leading the way, being the early adopters, driving down the price, and opening the door for the technology to find its way to other applications.

That's absolutely right. It's also true for what I've done in the past with digital musical instruments. Multi-touch devices began life as an electronic drum. That's why my group built the first publicly announced multi-touch device. It was a drum. That's all we were trying to make. And then it took on this whole other life 26 years later.

There's one other thing about what you just said that's really important and adds weight to games and recreational and creative uses of computers-the so-called "non-serious" things. They set expectations. When you're using technology that behaves in a way that delivers a fantastic experience in your "non-serious" digital activities and then walk into your workplace where you're doing "serious" stuff, it's like getting a bucket of cold water thrown over you.

The history of technology evolution has been that creative things that are loose and free and deliver a great experience push expectations and moves into the mainstream of technology. The physical technology we see in gaming right now will grow cost effective, but far more important is the body of experience of the people who use these things to great pleasure and a high quality of experience. Those expectations are going to be transferred over and applied to the workplace. That's where expectations will lead and technology will have to follow.

Part three of my interview with Bill Buxton will come later this week.

Follow me on Twitter: @ chadsapieha

Interact with The Globe