Learning Adversarial Grammars for Future Prediction

AJ Piergiovanni, Alexander Toshev, Anelia Angelova, Michael Ryoo

Keywords: future prediction, grammar

TL;DR: We design a grammar that is learned in an adversarial setting and apply it to future prediction in video.

Abstract:

In this paper, we propose a differentiable adversarial grammar model for future prediction. The objective is to model a formal grammar in terms of differentiable functions and latent representations, so that their learning is possible through standard backpropagation. Learning a formal grammar represented with latent terminals, non-terminals, and productions rules allows capturing sequential structures with multiple possibilities from data.

The adversarial grammar is designed so that it can learn stochastic production rules from the data distribution. Being able to select multiple production rules leads to different predicted outcomes, thus efficiently modeling many plausible futures. We confirm the benefit of the adversarial grammar on two diverse tasks: future 3D human pose prediction and future activity prediction. For all settings, the proposed adversarial grammar outperforms the state-of-the-art approaches, being able to predict much more accurately and further in the future, than prior work.