Unsupervised Intuitive Physics from Past Experiences

Sebastien Ehrhardt, Aron Monszpart, Niloy Mitra, Andrea Vedaldi

Abstract: We consider the problem of learning models of intuitive physics from raw, unlabelled visual input. Differently from prior work, in addition to learning general physical principles, we are also interested in learning ``on the fly'' physical properties specific to new environments, based on a small number of environment-specific experiences. We do all this in an unsupervised manner, using a meta-learning formulation where the goal is to predict videos containing demonstrations of physical phenomena, such as objects moving and colliding with a complex background. We introduce the idea of summarizing past experiences in a very compact manner, in our case using dynamic images, and show that this can be used to solve the problem well and efficiently. Empirically, we show, via extensive experiments and ablation studies, that our model learns to perform physical predictions that generalize well in time and space, as well as to a variable number of interacting physical objects.

Code: https://drive.google.com/file/d/1Lr4sB_WOlSQ5qAfBzkfqVBrYDXe9KgxE/view?usp=sharing

Keywords: Intuitive physics, Deep learning