Rule-Bottleneck RL: Learning to Decide and Explain for Sequential Resource Allocation via LLM Agents

ICLR 2026 Conference Submission13366 Authors

Keywords: Joint decision and explanation, rule-bottleneck, constrained-resource allocation, agent

TL;DR: We design a novel rule-based RL framework that provides joint explanation and decision optimization for high-stake resource allocation problems.

Abstract: Deep Reinforcement Learning (RL) has demonstrated remarkable success in solving sequential resource allocation problems, but often suffers from limited explainability and adaptability---barriers to integration with human decision-makers. In contrast, LLM agents, powered by large language models (LLMs), provide human-understandable reasoning but may struggle with effective sequential decision making. To bridge this gap, we introduce Rule-Bottleneck RL (RBRL), a novel LLM agent framework for resource allocation problems that jointly optimizes language-based decision policy and explainability. At each step within RBRL, an LLM first generates candidate rules---language statements capturing decision priorities tailored to the current state. RL then optimizes rule selection to maximize environmental rewards and explainability, with the LLM acting as a judge. Finally, an LLM chooses the action (optimal allocation) based on the rule. We provide conditions for RBRL performance guarantees as well as the finite-horizon evaluation gap of the learned RBRL policy. Furthermore, we provide evaluations in real-world scenarios, particularly in public health, showing that RBRL not only improves the performance of baseline LLM agents, but also approximates the performance of Deep RL while producing more desirable human-readable explanations. We conduct a survey validating the improvement in the quality of the explanations.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 13366