How do you build computer vision that runs in real time on constrained automotive hardware without compromising robustness? Neonode's Engineers, Anders Sahlström and Johan Villysson, explain that the answer lies in efficient network design, synthetic data, rigorous validation, and a development pipeline built for rapid iteration.
Automotive computer vision operates under demanding constraints. A Driver Monitoring System (DMS) must process camera images in real time, perform reliably across diverse drivers and conditions, and run on the hardware available in the vehicle. More compute is not always available, and more neural networks are not necessarily the solution.
For Anders Sahlström, Machine Learning Expert at Neonode, these challenges are familiar. Before working with machine learning, he developed analytical computer vision algorithms for industrial robots, where real-time execution and limited computing resources were already critical requirements. Automotive applications add another challenge: maintaining robust performance in highly unpredictable environments.
Johan Villysson, Software Engineer, brings more than a decade of automotive experience spanning active safety, autonomous driving, and driver and occupant monitoring systems.
Together, their work focuses on a core engineering problem: extracting as much useful information as possible from camera data while keeping the complete vision pipeline efficient enough to run on target hardware.
Model accuracy alone is not enough in automotive applications. A system must also meet strict timing requirements, operate within available compute resources, and behave predictably when conditions are less than ideal. As Anders explains, robustness means handling situations that are difficult to anticipate during development.
"You can never know what strange things people will do in the real world, what clothing they will wear, what props they will have, what actions they will perform, or what lighting conditions the system needs to operate under."
A model that performs well under familiar conditions may still fail when exposed to unseen combinations of variables. High performance therefore requires designing for variation from the outset.
Johan emphasizes that robustness and compute efficiency are inseparable. Even the most accurate model provides little value if it cannot run at the required speed on the vehicle's hardware.
This makes network architecture a critical design decision.
Rather than relying on a single, large neural network to handle multiple, distinct tasks, Neonode uses a modular approach, with compact, task-specific models designed to solve individual problems efficiently. Closely related functions, such as gaze estimation and head pose estimation, can be handled within the same model, while distinct tasks, such as seatbelt routing or phone-in-hand detection, are addressed by separate models. This approach allows each model to focus on its specific task, supporting efficient use of compute resources while maintaining the flexibility to combine multiple capabilities within the overall system.
The goal is to improve overall efficiency while avoiding the cumulative compute cost by running multiple specialized networks. At the same time, the architecture remains modular where tasks have fundamentally different requirements.
Not every computer vision function needs to run at the same frequency.
Functions such as gaze estimation and drowsiness detection require rapid updates, typically running at 30 frames per second, or approximately every 33 milliseconds. Other features, such as seatbelt routing analysis, do not require the same update rate.
This allows compute resources to be allocated according to the needs of each function rather than processing everything at the highest possible frequency. The result is a flexible architecture in which time-critical functions receive the processing resources they require, while less demanding tasks consume fewer resources.
Optimization, however, extends beyond the neural network itself. According to Anders, the entire execution path matters, including C++ implementation, data handling, memory management, and buffers. Network efficiency is only one part of the compute budget. Bottlenecks elsewhere in the pipeline can quickly negate gains achieved during model development.
Effective optimization therefore requires considering the complete chain, from camera input and preprocessing through inference and output.
Building efficient models is only part of the challenge. They must also generalize well without requiring impractical amounts of real-world training data. To achieve this, Neonode makes extensive use of synthetic data. Synthetic datasets allow engineers to generate controlled variations in facial appearance, head pose, camera position, lighting conditions, and occlusions. This provides broad coverage of scenarios that would be difficult, expensive, and time-consuming to collect in the real world.
Johan highlights the ability to generate large variations in facial characteristics. Building an equivalent real-world dataset would require recording many different people, which would be both costly and timely, while synthetic data can provide that diversity through rendering.
The same applies to camera geometry. Anders notes that synthetic datasets can include a wide range of camera positions that would be difficult to replicate through physical data collection alone.
For compute-constrained systems, this is particularly important. Models must generalize across different vehicle installations and operating conditions without becoming unnecessarily large or complex. However, synthetic data is not a substitute for real-world validation. Artificial images do not perfectly reproduce the characteristics of real camera data, creating what is commonly known as the 'sim-to-real domain gap'.
To address this, Neonode combines synthetic data with real-world recordings and carefully established ground truth. Synthetic data provides broad coverage and variation, while real-world data helps bridge the domain gap and expose limitations that require further improvement before release. The two sources complement each other rather than compete.
Reliable evaluation depends on reliable ground truth. Gaze estimation provides a good example. To determine accuracy, engineers must know where a person was actually looking, not simply compare predictions against other model outputs.
Neonode has developed a dedicated data collection setup that establishes this ground truth across different camera configurations. By reconstructing three-dimensional reference points, determining camera position, and mapping image observations into a 3D environment, the team can calculate reference gaze directions for both training and evaluation. This controlled setup provides a consistent way to compare performance across multiple camera placements and installations.
Real-world recordings add another critical element. They expose systems to unexpected combinations of lighting conditions, clothing, occlusions, eyewear, and driver behavior. The goal is not simply maximizing performance under ideal conditions, but understanding how performance changes when conditions deteriorate. For example, if glare obscures a driver's eyes, the system should not confidently report a gaze direction based on the eyes. Robustness includes managing uncertainty appropriately when visual information is limited.
The value of synthetic data becomes especially clear when testing reveals a specific weakness. If a certain edge case reduces performance, engineers can generate targeted synthetic examples, retrain the model, and compare results against real-world evaluation data.
Neonode's cloud-based training infrastructure allows multiple experiments to run in parallel, accelerating comparison of different training strategies and data augmentations.
Anders describes an example in which additional augmentation was introduced to improve performance with newly released bulky headsets with cross-face mircrophones. Within hours, multiple training runs could be completed and evaluated. The importance is not the exact case, but the ability to move rapidly from problem identification to testing and validation.
The same workflow can address challenges involving occlusions, different lighting effects, or other conditions discovered during real-world testing. Engineers identify the issue, generate or collect relevant data, train candidate solutions, and evaluate results against real-world datasets. This tight feedback loop allows improvements without redesigning the entire system.
Rapid iteration also depends on organizational structure. At Neonode, machine learning specialists, rendering engineers, C and C++ developers, testers, and data collection experts work closely together. This gives the team direct control over the full development pipeline which is entirely in-house, from generating training data to deploying optimized software.
According to Anders, this integrated approach allows specialists to collaborate directly when new data or changes are required.
Johan similarly highlights the value of having expertise from multiple disciplines readily available. He adds, “In previous organizations there was a lot of time spent navigating internal structures and getting decisions through different layers, rather than actually solving the engineering problem. I value the direct collaboration between disciplines at Neonode. When you identify an issue, the people who can solve it are right there, and you can move from problem to solution much faster.”
This agility is particularly important in automotive development, where customer requirements, hardware limitations, and real-world observations continually introduce new constraints. Optimization is rarely a single change. It is the result of countless decisions involving architecture, data generation, inference, implementation, and validation. Fast iteration makes it possible to explore alternatives and identify measurable improvements more efficiently.
Automotive applications are especially demanding because they combine limited computing resources with strict requirements for real-time operation and robustness. Hardware platforms may be constrained, while software must handle an enormous variety of environmental conditions and driver behaviors. Yet the underlying methods extend beyond driver monitoring.
Johan notes that the same combination of neural network design, synthetic data generation, and rapid training infrastructure can be applied to many other computer vision problems involving human activity and intent, including retail environments and human behavior analysis. The principles remain consistent: select the right architecture, generate representative data, validate against real-world conditions, and optimize the entire execution pipeline. The challenge is not simply building a more accurate neural network. It is building a complete computer vision system that delivers the required performance, at the required speed, on the available hardware.
For Anders and Johan, low compute, robustness, and development efficiency are interconnected engineering requirements. By designing the model, the data, and the software pipeline together, Neonode creates systems that are optimized not for benchmark performance alone, but for the realities of the real world.