Benchmark for Temporal, Ambiguous, and Grounded Embodied Question-Answering
Keywords: embodied question answering, episodic memory question answering, ambiguity
TL;DR: A new benchmark focused on complex spatially and temporally ambiguous questions in embodied environments.
Abstract: The problem of question ambiguity, while highlighted as an open issue, is often overlooked in the literature on Embodied Question Answering (EQA) and Episodic Memory Question Answering (EM-EQA). This paper proposes a structured approach to handle ambiguity in the egocentric data. Our benchmark, called TAG-EQA, utilizes spatial and temporal grounding to distinguish between objects, positions, and events and ensures that obtained structured answers preserve information fully while effectively resolving ambiguity. We introduce a new dataset, specifically designed for ambiguous grounded Episodic Memory QA. The dataset incorporates situated spatial reasoning, temporal conditions, and diverse visual features. Our new evaluation procedure tackles grounded natural language answers. It reveals that some of the most modern approaches still struggle with efficient information extraction and processing in ambiguous scenarios. We hope that TAG-EQA will serve as both a valuable tool for generating complex EM-EQA data and that the proposed evaluation benchmark will propel progress in agentic AI and embodied reasoning.
Primary Area: datasets and benchmarks
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 11634
Loading