Video Prediction with Variational Temporal Hierarchies

Vaibhav Saxena, Jimmy Ba, Danijar Hafner

Keywords: latent dynamics, temporal abstraction, video prediction, probabilistic modeling, variational inference, deep learning

Abstract: Deep learning has shown promise for accurately predicting high-dimensional video sequences. Existing video prediction models succeeded in generating sharp but often short video sequences. Toward improving long-term video prediction, we study hierarchical latent variable models with levels that process at different time scales. To gain insights into the representations of such models, we study the information stored at each level of the hierarchy via the KL divergence, predictive entropy, datasets of varying speed, and generative distributions. Our analysis confirms that faster changing details are generally captured by lower levels, while slower changing facts are remembered by higher levels. On synthetic datasets where common methods fail after 25 frames, we show that temporally abstract latent variable models can make accurate predictions for up to 200 frames.

One-sentence Summary: We introduce and investigate the properties of a temporally-abstract latent dynamics model, trained using a variational objective, for long-horizon video prediction.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=VM_HTBp3ec