Cookie settings
Insights into Neonode’s MultiSensing® Neural Networks
Richard Ek, Product Development Engineer, provides a deep-dive into how Neonode’s MultiSensing® technology leverages advanced neural networks to boost safety in Driver Monitoring Systems.
Driver monitoring systems are crucial for road safety, allowing vehicles to observe driver behavior and intervene when needed. The system uses information from in-cabin cameras and sophisticated software then interprets the image data to differentiate between safe driving and potentially dangerous behaviors like distraction or drowsiness. A significant challenge in developing this software is the inherent complexity of human behavior. Furthermore, the system must constantly adapt to dynamic conditions such as varying light, driver attire (glasses or hats) and diverse differences in human features.
For the software to consistently deliver reliable results, an AI model must first be established using a robust neural network architecture. The model is fed with input data, in the case of driver monitoring software - images of drivers, and it is trained on the specific actions or objects in the images. The neural networks are made up of many computational neurons, which are layered and extract different information from the data, for example, the first hidden layers is responsible for low level signals, such as edges, shapes or boundaries, while the subsequent layers performs more complicated tasks like identifying objects or actions. In the last layer the system outputs its interpretation based on the model’s learnings, such as detecting signs of drowsiness.
To learn more about neural networks, we spoke to Richard Ek, Product Development Engineer, who shares his expertise in developing Neonode’s driver monitoring software, MultiSensing®, to ensure speed and reliability. In this discussion, Richard offers insights into the methodologies, challenges and success strategies for creating robust models for real-world driving scenarios.
How are the Neonode neural network designs used to enhance driver safety in MultiSensing for real-world driving scenarios?
Neonode’s neural networks improve driver safety by processing live in-cabin camera feeds. The system can detect when a driver is drowsy, looking away from the road, or, for example, has their hands off the wheel. Once detected, the system can trigger alerts or vehicle interventions, giving the driver time to correct their behavior and avoid accidents.
Can you describe the end-to-end pipeline for a DMS neural network, from data collection to deployment on an automotive ECU (Electronic Control Unit).
The pipeline involves:
Synthetic data generation – Specifying distribution ranges for parameters and sending the job to the render farm.
Data Preprocessing – Preparing the images and annotations for the dataset pipeline and adding augmentations.
Model Training –Using deep convolutional architectures, the network gradually learns to recognize visual patterns.
Validation & Testing – Evaluating accuracy and robustness on separate validation datasets.
Optimization – Quantization, pruning, and other techniques to reduce computational load without sacrificing accuracy.
Deployment – Integrating the optimized model onto the automotive ECU by converting it into a lightweight, hardware-compatible format.
Continuous Monitoring – Gathering post-deployment feedback for iterative improvements.
What are the benefits of using synthetic data to train neural networks, and can you explain Neonode’s synthetic training data pipeline?
Synthetic data allows for unlimited, perfectly labeled datasets. Our pipeline uses Maya to create realistic driver avatars and environments, varying parameters such as lighting, facial accessories and head positions. The render farm generates large volumes of images ensuring coverage of edge cases and improving model generalization.
How does Neonode ensure the robustness of a neural network against challenging conditions like low light, occlusions (e.g., sunglasses) or adversarial inputs?
This is accomplished by including challenging scenarios such as low light or partial occlusions in the synthetic datasets. The team also applies data augmentation techniques, like adding simulated glare, shadows and occluding objects, so the model learns to handle these conditions reliably in the real-world.
DMS requires real-time processing on resource-constrained automotive hardware. How do you optimize neural networks to meet latency and computational constraints?
We optimize our neural networks by using techniques such as quantization, which reduces the precision of the computations while maintaining accuracy. By designing lightweight model architectures like MobileNet that are specifically tailored for embedded hardware.
How does Neonode’s neural networks adjust to different vehicle models with varying camera setups and cabin configurations?
Neonode designs its models to be camera-agnostic by training on a wide variety of camera perspectives and resolutions. Calibration parameters for each vehicle model are integrated at deployment, allowing for geometric and optical adjustments without retraining the core model.
Some driver monitoring solutions can lack reliability due to biases in the datasets, for example, they fail to identify certain demographics. How has Neonode overcome biases in the training data for MultiSensing to ensure consistently fair and accurate monitoring?
A major advantage of using synthetic data is that we can fully control the distributions of all parameters, such as demographics, to ensure that every group is represented. By systematically generating training data that includes a wide variety of ages, genders, ethnicities and facial features, the model is less likely to inherit the kinds of biases that often occur in purely real-world datasets.
In machine learning, overfitting can occur when a model learns the training data too well, leading to reduced performance when presented with new, unseen data. How does the team identify and mitigate overfitting in neural networks for driver monitoring?
Overfitting is identified through performance drops on unseen validation data and monitoring training vs. validation loss curves. We use mitigation strategies including dropout layers, early stopping and expanding dataset variety with more augmentation or more examples.
How do you stay updated on advancements in neural networks as well as the automotive regulatory mandates?
The team participates in industry conferences and follows key AI research publications. We have lunch-n-learn sessions as well so we share information across the company.
Want to learn more about Neonode’s Driver Monitoring Software? Click here or book a meeting with a consultant.
Profile:
Richard is an accomplished 3D artist with a distinguished academic background in computer graphics and programming from KTH Royal Institute of Technology and Nackademin in Stockholm, Sweden.
He oversees Neonode’s synthetic training data pipeline, managing render farm operations on Linux, creating and rigging 3D assets in Maya and developing TensorFlow datasets and training processes.
Richard Ek
Product Development Engineer