What can VLMs Do for Zero-shot Embodied Task Planning?

Xian Fu, Min Zhang, Jianye HAO, Peilong Han, Hao Zhang, Lei Shi, Hongyao Tang

Keywords: Vision Language Models, Embodied Task Planning

TL;DR: We propose an evaluation framework for VLMs for embodied task planning, along with a challenging benchmark, on which the experimental results indicate that GPT-4V is not yet a reliable task planner, and we offer many meaningful insights.

Abstract: Recent advances in Vision Language Models (VLMs) for robotics demonstrate their enormous potential. However, the performance limitations of VLMs for embodied task planning, which require high precision and reliability, remain ambiguous, greatly constraining their potential application in this field. To this end, this paper provides an in-depth and comprehensive evaluation of VLM performance in zero-shot embodied task planning. Firstly, we develop a systematic evaluation framework encompassing various dimensions of capabilities essential for task planning for the first time. This framework aims to identify the factors that constrain VLMs in producing accurate task plans. Based on this framework, we propose a benchmark dataset called ETP-Bench to evaluate the performance of VLMs on embodied task planning. Extensive experiments indicate that the current state-of-the-art VLM, GPT-4V, achieves only 19% accuracy in task planning on our benchmark. The main factors contributing to this low accuracy are deficiencies in spatial perception and object type recognition. We hope this study can provide data support and inspire more specific research directions for future robotics research.

Submission Number: 70