ARNOLD: A Benchmark for Language-Grounded Task Learning with Continuous States in Realistic Scenes

Ran Gong, Yizhou Zhao, Xiaofeng Gao, Jiangyong Huang, Qingyang Wu, Wensi Ai, Baoxiong Jia, Zhou Ziheng, Song-Chun Zhu, Siyuan Huang

Keywords: Grounded task learning, Continuous states, Simulated environment

TL;DR: Benchmark for grounded task learning for continuous object states in photo-realalistic scenes

Abstract: Understanding continuous object states and task goals is essential for task planning since they are generally not discrete in the real world. However, most previous task learning benchmarks assume discrete (\eg, binary) object states, making it barely applicable to transfer the policy from the simulated environment to the real world. Moreover, the trained robot's ability to follow human instructions based on grounding the actions and states is limited. To address such challenges, we present ARNOLD, a benchmark that evaluates language-grounded task learning with continuous states in realistic 3D scenes. ARNOLD consists of 8 language-conditioned manipulation tasks that require an in-depth understanding of continuous object states and policy learning for continuous goals. To encourage language-instructed learning, we provide template-generated demonstrations with language descriptions. We will benchmarked the task performances with state-of-the-art language-conditioned policy learning algorithms. We will release ARNOLD and host challenges to promote future research in embodied AI and robotics.