Failure-Aware Query Refinement for Reliable Open-Vocabulary Home-Robot Perception

Daun Jeong, Sohyeon Kim, Jouwon Song, Kyeongbo Kong

Keywords: Query Refinement, Open-Vocabulary, Agent

Abstract: Home robots in open-ended indoor environments must localize user- or task-specified objects using imperfect perception tools. Frozen open-vocabulary detectors are attractive, but class-name queries can miss the target, produce weak detections, or fire on distractors. We study failure-aware query adaptation for a frozen detector, modifying only the text query at test time. We propose Failure-Aware Multi-Agent Query Refinement, which diagnoses detector responses into GOOD, MISS, WEAK, and NOISY states, routes failures to state-specific query agents, and re-evaluates candidates by re-running the detector. On 9,740 ODSR-IHS image--target pairs, our method improves YOLOE from 0.6513 to 0.7667 AP50 and from 0.6129 to 0.7504 Top-1 accuracy, with the largest gains in WEAK and NOISY cases.

Track: Short Paper (4 pages)

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 316