Coarse Correspondences Boost 3D Spacetime Understanding in Multimodal Language Model
Keywords: Multimodal Language Model; 3D Understanding; Temporal Understanding
Abstract: Multimodal language models (MLLMs) are increasingly being applied in real-
world environments, necessitating their ability to interpret 3D spaces and compre-
hend temporal dynamics. Current methods often rely on specialized architectural
designs or task-specific fine-tuning to achieve this. We introduce COARSE CORRE-
SPONDENCES, a simple lightweight method which enhances MLLMs’ understand-
ing of 3D and temporal concepts using only 2D images, without modifying the
architecture or task-specific fine-tuning. Our method uses a lightweight tracking
model to identify primary object correspondences between frames in a video or
across different image viewpoints, and then conveys this information to MLLMs
through visual prompting. We demonstrate that this simple training-free approach
brings substantial gains to GPT4-V/O consistently on four benchmarks that require
3D and temporal understanding, including +20.5% improvement on ScanQA,
+9.7% on OpenEQA’s episodic memory subset, +6.0% on the long-form video
benchmark EgoSchema, and +11% on the R2R navigation benchmark. Addition-
ally, we show that COARSE CORRESPONDENCES can also enhance open-source
MLLMs’ understanding of 3D space (by +6.9% on ScanQA) when applied in both
training and inference and that the improvement can generalize to unseen datasets
such as SQA3D (+3.1%). Taken together, we show that COARSE CORRESPON-
DENCES effectively and efficiently boosts models’ performance on downstream
tasks requiring 3D and/or temporal understanding.
Supplementary Material: zip
Primary Area: foundation or frontier models, including LLMs
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12639
Loading