Towards Human-Like Instruction Navigation in Real-World

Jiseon Kim, Daeun Song, Bokeon Suh, HyoseokJu, Yumin Lee, Xuesu Xiao, Giseop Kim

Keywords: Navigation Dataset, Socially-Aware Navigation, Human-Robot Interaction (HRI), Object Goal Navigation (OGN), Commonsense Reasoning

Abstract: For robots to successfully accomplish tasks in human daily life, they must possess the capability to navigate by understanding human-like instructions that people often use. However, current Object Goal Navigation (OGN) research primarily relies on detailed instructions that are not typically used in the wild, and thus still lacks the ability to navigate with abstract human guidance. Prior real-world socially compliant navigation work focuses on executing explicit social-norm based instructions, without wayfinding instructions or an explicit goal. We present Human-like INsTruction–grounded Navigation (HINT), a novel task in which an embodied agent must reach a goal solely from human verbal and non-verbal instructions. To support this task, we construct SocialACT, the first real-world dataset that bridges the gap between abstract human instructions and robot behaviors. Unlike traditional OGN tasks that only assess episode-level success or failure, SocialACT enables progress-based evaluation at sub-instruction granularity.

Submission Number: 9