HMI Design for Safer Takeover From Autonomous to Manual Driving

Semi-autonomous vehicles are transforming drivers into passengers. We take a look at smart HMI ideas for safe handovers from autonomous to manual driving, and examine how factors such as cognitive load and experience can effect driver takeover times.

With highly automated vehicles entering the market, there is a fundamental shift in the way the driver and the car interact. As the level of autonomy increases, the driver will take on an observer or passenger role for larger portions of the drive.

Some key challenges that this shift presents is how to safely hand over control from automated to manual mode, how to keep the driver feeling safe when they’re not in control of the vehicle and how the car can cater to the driver’s changing needs when they no longer drive. This has resulted in new demands for intelligent human-to-machine interaction (HMI) solutions, sophisticated driver monitoring and even refined vehicle interiors.

Taking Back Control

During a takeover, whether by choice or as commanded by the vehicle, the driver simultaneously needs to cognitively assess the environment and traffic situation, as well as physically reposition themselves to a driving state (putting their hands on the steering wheel and feet on the pedals). For a successful takeover, the car needs to have insight into the driver’s readiness before switching off the autonomous mode, as the takeover time can vary greatly for each driver and on each occasion. There are many studies on how different factors can influence the takeover performance of a driver. Key factors include cognitive load, engagement in non-driving-related tasks (NDRTs), driving style and driver experience.

Because there are so many influencing factors, finding a takeover strategy that works for every driver in every situation has proved to be difficult. From the studies performed it appears that there is no single time budget (time needed to respond to a warning) which can meet the requirements of all different scenarios. An optimal takeover process therefore needs to be designed in a way which it can identify the influencing factors and tailor the takeover in each scenario.

So how can the vehicle understand the state of the driver and adapt accordingly?

Cognitive Load and NDRTs

Cognitive load, both excessively high load, such as when the driver is engaged with a NDRT that requires their full concentration, and excessively low load, such as when the driver is resting, can increase takeover times and reduce the quality of takeover maneuvers. The best performance is obtained when the driver is engaged in an activity requiring moderate concentration, such as watching an entertaining video. Measuring cognitive load is however a complex endeavor. While studies have shown a strong relation between blink frequency and cognitive load, the nature of this relation depends on the type of task performed. For demanding visual tasks, such as tracking an object with your eyes, the eye blink rate decreases. On the other hand, tasks which do not need visual attention, the relation is the opposite. An increase in cognitive load due to an auditory task will result in an increased blink rate . These contradicting results show that simply monitoring the drivers eye behavior is not adequate enough really understand and adjust the handover to meet the cognitive state of the driver. The vehicle requires a deeper understanding of the situation at hand.

Driving Style and Experience

Recent studies have also found that driving style impacts driver takeover performance. How drivers operate the vehicle in manual mode reflects how they act during an automated driving takeover scenario. For example, drivers who keep long headways to the car in front during manual driving are also more cautious when taking over from the automated system. Another study comparing defensive and aggressive drivers found that defensive drivers generally perform better during a takeover than their aggressive counterparts, and so the optimal warning strategies differ between the two types. Driver experience also plays a role, with less experienced drivers exhibiting poorer takeover performance.

Understanding Human State

Understanding the state of the driver and having the vehicle adapt accordingly first requires intelligent analysis of the driver and cabin, and then seamless communication to other systems in the vehicle. Neonode’s driver monitoring solution can play a crucial role in understanding and predicting the response of the driver.

Built upon Neonode’s proprietary MultiSensing® technology, our driver monitoring software can detect if the driver is resting or sleeping by analyzing not only the driver's eyelids, but combining this data point with posture and steering wheel engagement. The ADAS can use this data to determine if more time is needed to reach physical readiness.

A longer time frame may also be needed if the driver is occupied with NDRTs. MultiSensing can see what the driver is doing with their hands, for example, holding a mobile phone, while simultaneously scanning the driver’s gaze to ensure that the driver has perceived the current traffic situation.

Advanced driver monitoring systems also open the way for driver identification. By collecting data from other systems such as braking behavior, acceleration patterns, speed and the common headway distance used by the driver, each driver can have a personalized ADAS profile based on their individual driving style, which can be applied from the moment they enter the vehicle.