Efficient world models with tree-structured sparsity

Reza Sedghi, Marcel Nieveler, Anand Subramoney, David Kappel, Robin Schiewer

Keywords: world models, sparsity, efficient computation

TL;DR: Tree-structured Fast Feedforward layers make transformer world models substantially sparser and faster while preserving predictive quality, improving their suitability for resource-constrained robots.

Abstract: Reliable robot autonomy requires internal models that capture semantic structure and temporal consequences of actions, yet such world models are often too computationally expensive for on-robot deployment. We study tree-structured Fast Feedforward (FFF) layers to sparsify transformer world models without substantially degrading predictive quality. FFF replaces dense feed-forward blocks by hard-routed hierarchical computation, thereby reducing the active computation per token. Building on prior evidence that FFF preserves performance in large transformer models under high sparsity, we evaluate a dense world-model baseline against FFF-based variants and analyse the trade-off between prediction quality and computational efficiency. Our results suggest that structured sparsity is not only a model compression technique, but also an enabling mechanism for executing richer semantic prediction and control directly on resource-constrained robots. This positions sparse world models as a promising ingredient for reliable robot autonomy.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 8