ABBEL: LLM Agents Acting through Belief Bottlenecks Expressed in Language

Aly Lidayan, Jakob Brandt Bjorner, Satvik Golechha, Alane Suhr

Keywords: multi-step, RL, exploration, memory, reasoning, beliefs, context management

Abstract: As the length of multi-step interactive language tasks increases, it becomes computationally impractical to keep full interaction histories in context. We propose a general and interpretable approach: Acting through Belief Bottlenecks Expressed in Language (ABBEL), which replaces long multi-step interaction history by a belief state, i.e., a natural language summary of what has been discovered about task-relevant unknowns. Under ABBEL, at each step the agent first updates the prior belief with the most recent observation from the environment, then uses only the updated posterior belief to select an action. We systematically evaluate frontier models under ABBEL across six diverse multi-step environments, finding that (1) ABBEL significantly reduces context lengths, enabling near-constant memory use over interaction steps, (2) the generated beliefs are interpretable, and (3) bottlenecks can reduce unnecessary reasoning. However, it is challenging to generate beliefs that are both concise and sufficient, and in some environments we observed inferior performance due to discarding valuable information or belief update errors. Motivated by this, we show that Reinforcement Learning is effective for improving the ability of LLM agents to generate and reason through belief bottlenecks. Training Qwen2.5-7B-Instruct under both ABBEL and full history settings, ABBEL quickly catches up with a 40% increase in performance while maintaining near-constant belief lengths over interaction steps.

Submission Type: Research Paper (4-9 Pages)

Submission Number: 60