From Actions to Words: Towards Abstractive-Textual Policy Summarization in RL

Sahar Admoni, Assaf Hallak, Yftah Ziser, Omer Ben-Porat, Ofra Amir

Keywords: Explainable RL, Policy Summarization, Human-Agent Interaction

TL;DR: The paper advocates a new paradigm of abstractive-textual policy explanations to make RL agent behavior more understandable.

Abstract:

Explaining reinforcement learning (RL) agents remains challenging, as policies emerge from complex reward structures and neural representations that are difficult for humans to interpret. Existing approaches rely on curated demonstrations that reveal local behaviors but offer limited insight into global strategy, leaving users to infer intent from raw observations.
We propose SySLLM (Synthesized Summary using Large Language Models), a framework that reformulates policy interpretation as a language-generation problem. Rather than relying on visual demonstrations, SySLLM translates spatiotemporal trajectories (an input modality outside the natural domain of LLMs) into structured text and prompts the model to produce coherent natural-language summaries that describe the agent’s goals, exploration style, and decision patterns.
SySLLM scales to long-horizon and semantically rich environments without task-specific fine-tuning, leveraging the world knowledge and compositional reasoning of LLMs to capture latent behavioral structure across diverse agents. Expert evaluations show that SySLLM summaries align closely with human analyses of policy behavior, and in a large-scale user study, 75.5% of participants preferred these textual summaries over state-of-the-art demonstration-based explanations.

Area: Human-Agent Interaction (HAI)

Generative A I: I acknowledge that I have read and will follow this policy.

Submission Number: 370