Ask more, know better: Reinforce-Learned Prompt Questions for Decision Making with Large Language Models

Xue Yan, Yan Song, Xinyu Cui, Filippos Christianos, Haifeng Zhang, David Henry Mguni, Jun Wang

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Large language models, reinforcement learning, machine learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a new leader-follower bilevel framework that is capable of generating its own prompts and then performing complex reasoning for guiding actions executed by an action-policy

Abstract:

Large language models (LLMs) demonstrate their promise in tackling complicated
practical challenges by combining action-based policies with chain of thought
(CoT) reasoning. Having high-quality prompts on hand, however, is vital to the
framework’s effectiveness. Currently, these prompts are handcrafted utilising
extensive human labor, resulting in CoT policies that frequently fail to generalise.
Human intervention is also required in order to develop grounding functions that
ensure low-level controllers appropriately process CoT reasoning. In this paper, we
take the first step towards a fully integrated end-to-end framework for task-solving
in real settings employing complicated reasoning. To that purpose, we offer a new
leader-follower bilevel framework capable of learning to ask relevant questions
(prompts) and subsequently undertaking reasoning to guide the learning of actions
to be performed in an environment. A good prompt should make introspective
revisions based on historical findings, leading the CoT to consider the anticipated
goals. A prompt-generator policy has its own aim in our system, allowing it to
adapt to the action policy and automatically root the CoT process towards outputs
that lead to decisive, high-performing actions. Meanwhile, the action policy is
learning how to use the CoT outputs to take specific actions. Our empirical data
reveal that our system outperforms leading methods in agent learning benchmarks
such as Overcooked and FourRoom.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7085