Evaluating World Models with LLM for Decision Making
Abstract: World model emerges as a key module in decision making, where MuZero and Dreamer achieve remarkable successes in complex tasks. Recent work leverages Large Language Models (LLMs) as general world simulators to simulate the dynamics of the world due to their generalizability. LLMs also serve as the world model for deliberative reasoning in Reasoning via Planning (RAP) and Tree of Thought (ToT). However, the world model is either evaluated as a general world simulator or as a functional module of the agent, i.e., predicting the transitions to assist the planning. In this work, we propose a comprehensive evaluation of the world models with LLMs from the decision making perspective. Specifically, we leverage the 31 diverse environments from (Wang et al., 2023; 2024) and curate the rule-based policy of each environment for the diverse evaluation. Then, we design three main tasks, i.e., policy verification, action proposal, and policy planning, where the world model is used for decision making solely. Finally, we conduct the comprehensive evaluation of the advanced LLMs, i.e., GPT-4o and GPT-4o-mini, on the environments for the three main tasks under various settings. The key observations include: i) GPT-4o significantly outperforms GPT-4o-mini on the three main tasks, especially for the tasks which require the domain knowledge, ii) the performance of the world model with LLM will be decreased for long-term decision-making tasks, and iii) the combination of different functionalities of the world model will brings additional unstabilities of the performance.
Submission Length: Regular submission (no more than 12 pages of main content)
Previous TMLR Submission Url: https://openreview.net/forum?id=epdB3yJz6t
Changes Since Last Submission: The comments of our previous submission from Action Editor focus on (a) a very high number of grammar errors, and (b) extensive use of undefined jargon. The paper is unfortunately desk-rejected by the Action Editor due to the two main reasons.
In this version, we carefully revise the texts of the paper. Specifically,
1. We rewrite the introduction section (i.e., Section 1), including fixing the grammar errors and explaining the technical terms used. We also revise other sections to make this paper more readable.
2. We also add Figure 1 to illustrate the motivation of this paper to facilitate the understanding.
3. We also add the discussion about the limitations of this work in Section 7.
We hope these changes can help both the editor and the reviewers to understand our motivations and contributions.
Assigned Action Editor: ~Marlos_C._Machado1
Submission Number: 3740
Loading