DISENTANGLED STATE SPACE MODELS: UNSUPERVISED LEARNING OF DYNAMICS ACROSS HETEROGENEOUS ENVIRONMENTS

Ðorđe Miladinović, Waleed Gondal, Bernhard Schölkopf, Joachim M. Buhmann, Stefan Bauer

Keywords: State Space Models, Sequential Data, Bayesian Filtering, Amortized Variational Inference, Disentangled Representations, Video Analysis

TL;DR: DISENTANGLED STATE SPACE MODELS

Abstract: Sequential data often originates from diverse environments. Across them exist both shared regularities and environment specifics. To learn robust cross-environment descriptions of sequences we introduce disentangled state space models (DSSM). In the latent space of DSSM environment-invariant state dynamics is explicitly disentangled from environment-specific information governing that dynamics. We empirically show that such separation enables robust prediction, sequence manipulation and environment characterization. We also propose an unsupervised VAE-based training procedure to learn DSSM as Bayesian filters. In our experiments, we demonstrate state-of-the-art performance in controlled generation and prediction of bouncing ball video sequences across varying gravitational influences.