Understanding image motion with group representations

Andrew Jaegle, Stephen Phillips, Daphne Ippolito, Kostas Daniilidis

Abstract: Motion is an important signal for agents in dynamic environments, but learning to represent motion from unlabeled video is a difficult and underconstrained problem. We propose a model of motion based on elementary group properties of transformations and use it to train a representation of image motion. While most methods of estimating motion are based on pixel-level constraints, we use these group properties to constrain the abstract representation of motion itself. We demonstrate that a deep neural network trained using this method captures motion in both synthetic 2D sequences and real-world sequences of vehicle motion, without requiring any labels. Networks trained to respect these constraints implicitly identify the image characteristic of motion in different sequence types. In the context of vehicle motion, this method extracts information useful for localization, tracking, and odometry. Our results demonstrate that this representation is useful for learning motion in the general setting where explicit labels are difficult to obtain.

TL;DR: We propose of method of using group properties to learn a representation of motion without labels and demonstrate the use of this method for representing 2D and 3D motion.

Keywords: vision, motion, recurrent neural networks, self-supervised learning, unsupervised learning, group theory

Data: [HMDB51](https://paperswithcode.com/dataset/hmdb51), [KITTI](https://paperswithcode.com/dataset/kitti)