Neural Implicit Representations for Physical Parameter Inference from a Single Video

Florian Hofherr, Lukas Koestler, Florian Bernard, Daniel Cremers

Keywords: neural implicit representations, physics learning, video interpretation, physical parameter estimation

Abstract: Neural networks have recently been used to model the dynamics of diverse physical systems. While existing methods achieve impressive results, they are limited by their strong demand for training data and their weak generalization abilities. To overcome these limitations, in this work we propose to combine neural implicit representations for appearance modeling with neural ordinary differential equations (ODEs) in order to obtain interpretable physical models directly from visual observations. Our proposed model combines several unique advantages: (i) It is trained from a single video, and thus overcomes the need for large training datasets. (ii) The use of neural implicit representation enables the processing of high-resolution videos and the synthesis of photo-realistic imagery. (iii) The embedded neural ODE has a known parametric form that allows for the identification of interpretable physical parameters, and (iv) long-term prediction in state space. (v) Furthermore, the photo-realistic rendering of novel scenes with modified physical parameters becomes possible.

One-sentence Summary: A method combining neural implicit representations with a rich physical model to infer physical parameters from a single short video clip