D2E: An Autonomous Decision-Making Dataset involving Driver States and Human Evaluation of Driving Behavior

Zehong Ke, Yanbo Jiang, Yuning Wang, Hao Cheng, Jinhao Li, Jianqiang Wang

Abstract: With the advancement of deep learning technology, data-driven methods are increasingly used for decision-making in autonomous driving, and the quality of datasets greatly influences the model's performance. Although current datasets have achieved significant improvements in vehicle and environment data, emphasis on human-end data including the driver states and human evaluation is insufficient. In addition, existing datasets mainly consist of simple scenarios such as car following, resulting in low interaction levels. In this paper, we introduce the Driver to Evaluation dataset (D2E), a dataset for autonomous driving decision-making that covers a comprehensive process of vehicle decision-making, including data on driver states, vehicle states, environmental situations, and evaluation scores from human reviewers. Apart from regular agents and surrounding environment information, we not only collect human factor data such as first-person view videos, physiological signals, and eye-tracking data, but also gather subjective rating scores from 40 human volunteers. The dataset comprises both driving simulator scenes and real-world scenes, with high-interaction situations designed and filtered to ensure behavior diversity. After data organization, preprocessing, and analyzing, D2E contains over 1100 segments of interactive driving case data covering from human driver factor to evaluation results, supporting the development of data-driven decision-making.