Unsupervised Motion Flow estimation by Generative Adversarial Networks

Stefano Alletto, Luca Rigazio

Abstract: In this paper we address the challenging problem of unsupervised motion flow estimation. Under the assumption that image reconstruction is a super-set of the motion flow estimation problem, we train a convolutional neural network to interpolate adjacent video frames and then compute the motion flow via region-based sensitivity analysis by backpropagation. We postulate that better interpolations should result in better motion flow estimation. We then leverage the modeling power of energy-based generative adversarial networks (EbGAN's) to improve interpolations over standard L2 loss. Preliminary experiments on the KITTI database confirm that better interpolations from EbGAN's significantly improve motion flow estimation compared to both hand-crafted features and deep networks relying on standard losses such as L2.

TL;DR: In this paper we estimate the motion flow between consecutive frames unsupervisedly using generative adversarial networks trained for image interpolation and performing sensitivity analysis by backpropagation.

Keywords: Computer vision, Deep learning, Unsupervised Learning

Conflicts: unimore.it, uniud.it, panasonic.com