Model-Based Meta-Learning for Algorithm Discovery

Theo Wolf, Alexander David Goldie, Jarek Luca Liesen, Uljad Berdica, Mattie Fellows, Jakob Nicolaus Foerster

Keywords: Reinforcement Learning, Offline Reinforcement Learning, Meta-Reinforcement Learning, Algorithm Discovery, Meta-learning, Model-Based Reinforcement Learning

TL;DR: Meta-training Reinforcement Learning algorithms in world models grants considerable speedups, without compromising performance.

Abstract:

Reinforcement Learning (RL) algorithms are typically hand-crafted through a slow and iterative scientific process.
While meta-RL promises to automate algorithm discovery, research into meta-RL has been held back by the large computational requirements of simulating environments for meta-training.
In this work, we introduce Model-Based Meta-Learning (MBML), a novel approach that uses learned world models as an efficient alternative to environment simulation. We show that MBML matches the performance of traditional online meta-RL at a fraction of the time and compute, without needing perfect world models. By substantially reducing the computational cost of meta-training, MBML lowers the barrier to entry for meta-RL research and enables algorithm discovery at a larger scale.

Submission Number: 37