An Emergent Symbolic Representation of Space as a Bridge Between Language and Reinforcement Learning in Continuous Environments

Ziqi Ma, Sao Mai Nguyen, Philippe Xu

Keywords: emergent symbolic representation; hierarchical reinforcement learning; active imitation learning; LLM planner

TL;DR: LLMs can take advantage of the symbolic representation of the spatio-temporal abstraction learned online by a hierarchical reinforcement learning algorithm. We couple LLM and RL as planners by active choice with intrinsic motivation

Abstract: Large Language Models (LLMs) exhibit their potential for interacting with reinforcement learning (RL) agents, for instance as high-level planners. However, space representation still impedes their application for embodied agents. We tackle this problem by taking advantage of a discrete world representation learned online by reinforcement learning. Our proposed algorithm, SGIM-STAR is a hierarchical RL method where the top-level agent is augmented with a partition-wise, learning-progress–driven switch between a RL-based planner and an LLM planner. The agent builds a discrete reachability-based partition of space online and uses intrinsic motivation to query the LLM only when beneficial, defaulting to RL as planner otherwise. This yields usage/cost efficiency: the RL-based planner dominates early and the LLM is leveraged as the representation matures. In AntMaze, SGIM-STAR achieves the best and most stable success among STAR, LLM-only, and a non-partitioned adaptive variant, avoiding mid-training collapses while reducing LLM calls. The results demonstrate a practical fusion of LLMs taking advantage of emerging symbolic models of the environment for long-horizon tasks.

Submission Type: Long Research Paper (< 9 Pages)

Submission Number: 54