Driver stress detection via multimodal fusion using attention-based CNN-LSTM

Luntian Mou, Chao Zhou, Pengfei Zhao, Bahareh nakisa, Mohammad Naim Rastgoo, Ramesh Jain, Wen Gao

Abstract:

Stress has been identified as one of major contributing factors in car crashes due to its negative impact on driving
performance. It is in urgent need that the stress levels of drivers can be detected in real time with high accuracy
so that intervening or navigating measures can be taken in time to mitigate the situation. Existing driver stress
detection models mainly rely on traditional machine learning techniques to fuse multimodal data. However, due
to the non-linear correlations among modalities, it is still challenging for traditional multimodal fusion methods
to handle the real-time influx of complex multimodal and high dimensional data, and report drivers’ stress levels
accurately. To solve this issue, a framework of driver stress detection through multimodal fusion using attention
based deep learning techniques is proposed in this paper. Specifically, an attention based convolutional neural
networks (CNN) and long short-term memory (LSTM) model is proposed to fuse non-invasive data, including eye
data, vehicle data, and environmental data. Then, the proposed model can automatically extract features
separately from each modality and give different levels of attention to features from different modalities through
self-attention mechanism. To verify the validity of the proposed method, extensive experiments have been
carried out on our dataset collected using an advanced driving simulator. Experimental results demonstrate that
the performance of the proposed method on driver stress detection outperforms the state-of-the-art models with
an average accuracy of 95.5%.