PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning

Angel Villar-Corrales, Sven Behnke

TL;DR: We propose PlaySlot-- an object-centric video prediction model that infers object representations and latent actions from unlabeled video sequences and uses them to forecast future video frames.

Abstract:

Predicting future scene representations is a crucial task for enabling robots to understand and
interact with the environment. However, most existing methods rely on videos and simulations
with precise action annotations, limiting their ability to leverage the large amount of avail-
able unlabeled video data. To address this challenge, we propose PlaySlot, an object-centric
video prediction model that infers object representations and latent actions from unlabeled
video sequences. It then uses these representations to forecast future object states and video
frames. PlaySlot allows the generation of multiple possible futures conditioned on latent actions,
which can be inferred from video dynamics, provided by a user, or generated by a learned action
policy, thus enabling versatile and interpretable world modeling. Our results show that PlaySlot
outperforms both stochastic and object-centric baselines for video prediction across different environments. Furthermore, we show that our inferred latent actions can be used to learn robot behaviors sample-efficiently from unlabeled video
demonstrations. Videos and code are available on our project website.

Lay Summary: Robots need to understand what’s happening around them and guess what might happen next. Most current methods need lots of detailed instructions to learn this, which takes a lot of time and effort. Our method, called PlaySlot, learns just by watching videos---like how people can learn by observing. It figures out what the important objects are, how they move, and then imagines what could happen next. This helps robots learn to do tasks even when we don’t give them step-by-step directions. It’s more flexible, easier to use, and works better than other tools.

Link To Code: https://github.com/angelvillar96/PlaySlot

Primary Area: General Machine Learning->Representation Learning

Keywords: object-centric video prediction, object-centric learning, inverse dynamics learning, action planning

Submission Number: 4634