World Model Predictive Control with Multimodal Adaptation: Towards Artificial Dynamics Intelligence for General Robotics

Baha Zarrouki, Johannes Betz

Keywords: world models, model predictive control, robotics learning, dynamics learning, autonomous systems

Abstract: Traditional robotic control systems rely on modular sense-plan-act architectures or end-to-end deep learning, both of which struggle with seamless task transfer, real-time adaptation, and computational efficiency. In this work, we propose World Model Predictive Control (WMPC), a novel framework that integrates World Models with Model Predictive Control (MPC) principles. By leveraging pre-trained differentiable world models to predict system dynamics and optimize control actions, WMPC eliminates the need for extensive policy training, unlike reinforcement learning (RL)-based world models. Our approach unifies multimodal state representations, task-specific cost learning, and constraint-aware optimization within a receding horizon framework. Inspired by human motor control and learning, it integrates general scene understanding and basic dynamics estimation while fine-tuning actions through rapid interaction with the environment. Furthermore, elastic model updating balances short-term corrections—instantaneous reactive adjustments to new dynamics—with long-term knowledge retention, enabling memory-augmented fine-tuning and improved general skill proficiency.

Submission Number: 24