iMotion-LLM: Motion Prediction Instruction Tuning
Keywords: Trajectory Prediction, Conditional Trajectory Prediction
TL;DR: Integrating large language models with motion prediction modules to enable text-guided trajectory prediction.
Abstract: We introduce iMotion-LLM, a Multimodal Large Language Model (LLM) integrated with trajectory prediction, designed to guide interactive multi-agent scenarios. Unlike conventional multimodal trajectory prediction approaches, iMotion-LLM generates diverse and feasible future trajectories conditioned on textual instructions as a guidance signal. By augmenting real-world driving scenarios in the Waymo Open Motion Dataset (WOMD) with textual motion instructions, we propose InstructWaymo data augmentation. Leveraging this data augmentation, iMotion-LLM integrates a pretrained LLM, fine-tuned with LoRA, to map scene features into the LLM input space. Key results demonstrate that making the trajectory prediction model conditional improves its instruction-following capabilities. Specifically, the integration of the LLM enables a 11.07x ratio of actual-scenario feasible to infeasible recall instruction following, compared to 5.92x when using the Conditional GameFormer alone. These findings highlight the ability of iMotion-LLM to generate trajectories that not only align with feasible instructions but also reject infeasible ones, enhancing overall safety. Despite its improvements in instruction following, iMotion-LLM inherits the strong trajectory prediction performance of the baseline model, making it versatile across different driving modes. This combination of skills positions iMotion-LLM as a powerful augmentation technique for trajectory prediction models, empowering autonomous navigation systems to better interpret and predict the dynamics of multi-agent environments. This work lays the groundwork for future advancements in instruction-based motion prediction.
Primary Area: applications to robotics, autonomy, planning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 8063
Loading