ChatAni: Language-Driven Multi-Actor Animation Generation in Street Scenes

Abstract: Generating interactive and realistic traffic participant animations from instructions is essential for autonomous driving simulations. Existing methods, however, fail to comprehensively address the diverse participants and their dynamic interactions in street scenes. In this paper, we present ChatAni, the first system capable of generating interactive, realistic, and controllable multi-actor animations based on language instructions. To produce fine-grained, realistic animations, ChatAni introduces two novel animators: PedAnimator, a unified multi-task animator that generates interaction-aware pedestrian animations under varying task plans, and VehAnimator, a kinematics-based policy that generates physically plausible vehicle animations. For precise control through complex language, ChatAni employs a multi-LLM-agent role-playing approach, using natural language to plan the trajectories and behaviors of different participants. Extensive experiments demonstrate that ChatAni can generate realistic street scenes with interacting vehicles and pedestrians, benefiting tasks like prediction and understanding. All related code, data, and checkpoints will be open-sourced.

Submission Type: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=Z9fDUbm9iP&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DTMLR%2FAuthors%23your-submissions)

Changes Since Last Submission: In the previous submission, two reviewers raised no further concerns. However, one reviewer noted that Figures 3 through 8 lacked references to the corresponding quantitative experiments, and requested pseudocode for the overall pipeline. In this revised version, we have added the relevant quantitative experiment references to Figures 3-8. Additionally, we have included the pseudocode for the training and testing phases at the beginning of the supplementary material.

Assigned Action Editor: ~Aleksandra_Faust1

Submission Number: 8379