Generative World Models of Tasks: LLM-Driven Hierarchical Scaffolding for Embodied Agents

Brennen Hill

Keywords: Multi-agent reinforcement learning, embodied world models, hierarchical task networks, curriculum learning, robot soccer, language models, sim-to-real transfer, planning, neuro-symbolic AI, symbolic reasoning

TL;DR: This paper argues that for complex, long-horizon multi-agent tasks, the next frontier in embodied world models is to scale their structural complexity through hierarchical scaffolding, which provides a curriculum and makes learning tractable.

Abstract: Recent advances in agent development have focused on scaling model size and raw interaction data, mirroring successes in large language models. However, for complex, long-horizon multi-agent tasks such as robotic soccer, this end-to-end approach often fails due to intractable exploration spaces and sparse rewards. We propose that an effective world model for decision-making must model the world's physics and also its task semantics. A systematic review of 2024 research in low-resource multi-agent soccer reveals a clear trend towards integrating symbolic and hierarchical methods, such as Hierarchical Task Networks (HTNs) and Bayesian Strategy Networks (BSNs), with multi-agent reinforcement learning (MARL). These methods decompose complex goals into manageable subgoals, creating an intrinsic curriculum that shapes agent learning. We formalize this trend into a framework for Hierarchical Task Environments (HTEs), which are essential for bridging the gap between simple, reactive behaviors and sophisticated, strategic team play. Our framework incorporates the use of Large Language Models (LLMs) as generative world models of tasks, capable of dynamically generating this scaffolding. We argue that HTEs provide a mechanism to guide exploration, generate meaningful learning signals, and train agents to internalize hierarchical structure, enabling the development of more capable and general-purpose agents with greater sample efficiency than purely end-to-end approaches.

Submission Number: 46