RRdE: A Decision Making Framework for Language Agents in Interactive Environments

Xufeng Zhou, Linjing Li, Daniel Dajun Zeng

Abstract: Large language models(LLMs) have demonstrated remarkable planning and reasoning abilities, particularly as few-shot learners, when utilizing in-context learning. However, since LLMs are not grounded during training, they still encounter difficulties when acting as agents in tasks that require interaction with the environment, especially in scenarios that involve long-term and multistep interactions. Even when provided with a complete game trajectory as context, LLMs struggle to comprehend the meaning of each interaction step, and may easily hallucinate and fail. To address these challenges, we introduce the RRdE (Reasoning and Replanning during Exploration), a framework inspired by planning theories, designed for reasoning about actions and planning subgoals in complex interactive environments. The RRdE method can integrate the long-term planning ability and tooluse ability of LLMs, and transform the long-term sequential decision problem into a relatively simple reasoning problem, thereby reducing the error behavior caused by excessive context. We devise a reflection-based goal decomposition and replanning scheme, which enables the agent to overcome the strict sub-goal dependency problem caused by long-term goal planning. Consequently, RRdE achieves state-of-the-art performance in the few-shot learning setting in both AlfWorld and ScienceWorld environments, accomplishing 132 out of 134 test tasks in AlfWorld, and obtaining an average score of 82.16 in the 30 more complex and challenging scientific tasks in ScienceWorld, successfully completing 7 tasks with a full score of 100.