FoREST: Frame of Reference Evaluation in Spatial Reasoning Tasks
Keywords: Spatial language, Evaluation benchmark, Frame of reference
Abstract: Spatial cognition is one fundamental aspect of human intelligence. A key factor in spatial cognition is understanding the frame of reference (FoR) that identifies the perspective of spatial relations.
However, the AI research has paid very little attention to this concept.
Specifically, there is a lack of dedicated benchmarks and in-depth experiments analyzing large language models' (LLMs) understanding of FoR.
To address this issue, we introduce a new benchmark, **F**rame **o**f **R**eference **E**valuation in **S**patial Reasoning **T**asks (FoREST) to evaluate LLMs ability in understanding FoR.
We evaluate the LLMs in identifying the FoR based on textual context and employ this concept in text-to-image generation.
Our results reveal notable differences and biases in the FoR identification of various LLMs.
Moreover, the bias in FoR interpretations impacts the LLMs' ability to generate layouts for text-to-image generation.
To improve spatial comprehension of LLMs, we propose Spatial-Guided (SG) prompting, which guides the model in exploiting the types of spatial relations for a more accurate FoR identification.
The SG prompting improves the overall performance of FoR identification by alleviating their bias towards specific frames of reference.
Eventually, incorporating the FoR information generated by SG prompting in text-to-image leads to a more accurate visualization of the spatial configuration of objects.
Supplementary Material: zip
Primary Area: datasets and benchmarks
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12949
Loading