Unsupervised Hierarchical Video Prediction

Nevan Wichers, Dumitru Erhan, Honglak Lee

Abstract: Much recent research has been devoted to video prediction and generation, but mostly for short-scale time horizons. The hierarchical video prediction method by Villegas et al. (2017) is an example of a state of the art method for long term video prediction. However, their method has limited applicability in practical settings as it requires a ground truth pose (e.g., poses of joints of a human) at training time. This paper presents a long term hierarchical video prediction model that does not have such a restriction. We show that the network learns its own higher-level structure (e.g., pose equivalent hidden variables) that works better in cases where the ground truth pose does not fully capture all of the information needed to predict the next frame. This method gives sharper results than other video prediction methods which do not require a ground truth pose, and its efficiency is shown on the Humans 3.6M and Robot Pushing datasets.

TL;DR: We show ways to train a hierarchical video prediction model without needing pose labels.

Keywords: video prediction, visual analogy network, unsupervised, hierarchical

Data: [COCO](https://paperswithcode.com/dataset/coco)