Skip to main content

An employee drives a Tesla Model S electric car equipped with Autopilot hardware and software in Amsterdam on Oct. 27, 2015.Jasper Juinen/Bloomberg

The weak link when it comes to partially automated cars might be you.

Humans aren’t good at paying attention when it feels like there’s nothing much to do, in part because we’re too quick to trust computers to do things for us.

That presents a safety problem for current level 2 (partially automated) vehicles that combine lane-centering steering and adaptive cruise control. While such systems – including Tesla’s Autopilot and similar tech from Audi, BMW, Mercedes, Infiniti and others – likely help make cars safer, this technology is also introducing new dangers.

“The danger is that people might tune out from the task of driving because they get misled that the system is better than it actually is. That’s over-trust in a system, and there is research showing that problem does exist,” said Krzysztof Czarnecki, professor of electrical and computer engineering, and co-lead of autonomous driving research at the University of Waterloo.

Engaging Tesla’s Enhanced Autopilot or BMW’s Driving Assistant Plus, you immediately feel the steering start to move in your hands, keeping the car in the middle of its lane. When traffic slows down ahead, your car automatically slows down, leaving a safe distance. It feels good.

“You get this feeling that the car is driving itself. But, in reality, the system is fairly limited in terms of what it can understand of the environment,” Czarnecki said.

These level 2 systems are meant to complement a human driver, acting as a backup rather than a replacement. The person behind the wheel is, ultimately, totally responsible for controlling the vehicle.

“The first fatal accident [involving a partially automated car], in Florida with Joshua Brown, shows there will be people who tune out, and it has fatal consequences,” said Czarnecki.

Over-trust in cars was an issue even before lane-centering steering and adaptive cruise control. Incidences of drivers ending up in a lake because they blindly followed GPS directions are more common than you might imagine. It was even spoofed in an episode of The Office in which Steve Carell’s character, following the GPS, shouts, “It knows where it’s going!” as water splashes over the car’s windshield.

Not just cars

This problem isn’t unique to the emerging autonomous vehicle field either; it’s an issue the aerospace and robotics industries are tackling as well.

In a study by the Georgia Institute of Technology, people were asked to interact with a way-finding robot in a non-emergency situation. Then, in a simulated panic scenario with smoke and fire alarms blaring, “all 26 participants followed the robot in the emergency, despite half observing the same robot perform poorly in a navigation guidance task just minutes before.”

“Even when the robot pointed to a dark room with no discernible exit, the majority of people did not choose to safely exit the way they entered,” the study found.

“Many aspects of trust in autonomous systems are not well understood, and over-trust in them poses serious and numerous risks,” said Alan Wagner, assistant professor of aerospace engineering at Penn State University. His study, recently funded by the U.S. Air Force Office of Scientific Research, is exploring how we might make machines that take into account how their actions affect an operator’s trust.

The U.S. military is looking into autonomous landing systems for its helicopters, according to Wagner. It could help pilots in low-visibility conditions, but, he says, “we must be vigilant to prevent over-trust of such systems, which could lead to situations in which pilots fail to monitor the landing process or eventually lose the skill to land altogether.”

Driver monitoring

Once we get to fully autonomous, completely trustworthy consumer vehicles (not any time soon), this issue will be moot. Until then, better driver-monitoring systems (DMS) are needed to mitigate the danger of inattention due to boredom and over-trust in partially automated vehicles.

All level-2 cars have driver-monitoring systems that detect when a driver has his or her hands on the steering wheel. If you take them off the wheel for more than a couple seconds, the cars will warn you to put your hands back on.

“Just having a hands-on-the-steering-wheel detector is not enough,” said Czarnecki.

Indeed, a handful of idiots are life-hacking their way around the hands-on requirement by placing a water bottle on the steering wheel. (See YouTube for evidence.) The weight apparently fools the system into thinking you’re paying attention. Why would you trust a water bottle to drive your car? I can’t fathom. Please do not try this. Nobody wants to share the road with you and your water bottle co-pilot.

“Something like [Cadillac’s] Super Cruise is really good,” Czarnecki said, “because it actually monitors the driver’s eyes and direction the driver is looking, so it has a much better idea of what the driver is doing.”

The Cadillac system, as seen on the CT6 sedan, is so far the only car that uses an infrared camera, which can monitor the driver even in the dark. Systems like this have the potential to save lives.

The safety driver of an autonomous Uber test vehicle was streaming a TV show on her phone just before a crash that killed a pedestrian, according to the police report. An eye-tracking DMS would have noticed the safety driver was looking down a lot and would have sounded alarms, potentially saving the life of the pedestrian, Elaine Herzberg.

However, even the most advanced DMS won’t prevent a driver from mentally tuning out, as in the cases of those crashes when drivers say, "I was looking but I didn’t see the bicycle/car/truck."

It’s important to remember that, despite the glossy advertising and media hype, automated and autonomous vehicle technology is still in its infancy.

Next time you get in a car and turn on Autopilot, Pro Pilot, Piloted Driving, Co-Pilot 360 or any other similar driving assistant, you’ll be safest if you don’t trust it.