LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence

Zhuoling Li, Xiaogang Xu, Zhenhua Xu, Ser-Nam Lim, Hengshuang Zhao

TL;DR: We introduce the large auto-regressive model and referee reinforcement learning algorithm.

Abstract: Recent embodied agents are primarily built based on reinforcement learning (RL) or large language models (LLMs). Among them, RL agents are efficient for deployment but only perform very few tasks. By contrast, giant LLM agents (often more than 1000B parameters) present strong generalization while demanding enormous computing resources. In this work, we combine their advantages while avoiding the drawbacks by conducting the proposed referee RL on our developed large auto-regressive model (LARM). Specifically, LARM is built upon a lightweight LLM (fewer than 5B parameters) and directly outputs the next action to execute rather than text. We mathematically reveal that classic RL feedbacks vanish in long-horizon embodied exploration and introduce a giant LLM based referee to handle this reward vanishment during training LARM. In this way, LARM learns to complete diverse open-world tasks without human intervention. Especially, LARM successfully harvests enchanted diamond equipment in Minecraft, which demands significantly longer decision-making chains than the highest achievements of prior best methods.

Lay Summary: This paper proposes the referee reinforcement learning algorithm, which can fine-tune a large language model into a promising embodied policy, termed as large auto-regressive model in this work.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Primary Area: Reinforcement Learning->Planning

Keywords: large auto-regressive model, referee reinforcement learning, embodied intelligence

Submission Number: 1145