Language-Guided Traffic Simulation and Rare Event Synthesis for an Urban Intersection

Chengbo Zang, Wangshu Zhu, Mehmet Kerem Turkcan, Siyi Liu, Hanlin Wang, Xudong Chen, Boshra Khalili, Noshin Saiyara Ahmad, Peter J Jin, Javad Ghaderi, Gil Zussman, Zoran Kostic

Keywords: traffic simulation, rare event synthesis, large language model, agentic AI

TL;DR: We propose a data synthesis pipeline for generating realistic traffic scenes and safety-critical rare events under natural language instructions while providing agent relation annotations.

Abstract: We propose a data synthesis pipeline for generating realistic traffic scenes and safety-critical rare events under natural language instructions while providing agent relation annotations. The pipeline structurally comprises the scene planner, agent generator, waypoint filter, event reasoner, and trajectory refiner, while incorporating a language model backend for controlled inference. By decomposing high-level semantic reasoning and low-level scene execution, our framework is able to produce physically grounded agent trajectories that satisfy the social relation specifications. The pipeline is used to generate a dataset of 400 traffic scenes based on an urban traffic intersection, featuring agent relations such as collision and yielding, which are safety-critical but challenging to specify in real world traffic data. We evaluate the quality of the synthesized agent trajectories by the simulation-to-reality gaps, where the pipeline achieves an 84\% of instruction satisfaction rate equipped with the Claude3.5-Sonnet backend. We further showcase the usage of the synthesized dataset by testing traffic scene perception and precognition using a simple agentic pipeline, both outperforming non-LLM baselines by a noticeable margin.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 19