Incorporating physics into data-driven computer vision

Achuta Kadambi, Celso de Melo, Cho-Jui Hsieh, Mani B. Srivastava, Stefano Soatto

Abstract: Many computer vision techniques infer properties of our physical world from images. Although images are formed through the physics of light and mechanics, computer vision techniques are typically data driven. This trend is mostly performance related: classical techniques from physics-based vision often score lower on metrics compared with modern deep learning. However, recent research, covered in this Perspective, has shown that physical models can be included as a constraint into data-driven pipelines. In doing so, one can combine the performance benefits of a data-driven method with advantages offered from a physics-based method, such as intepretability, falsifiability and generalizability. The aim of this Perspective is to provide an overview into specific approaches for integrating physical models into artificial intelligence pipelines, referred to as physics-based machine learning. We discuss technical approaches that range from modifications to the dataset, network design, loss functions, optimization and regularization schemes. Although computer vision techniques are often data-driven, they can be enhanced by including the physical models underlying image formation as constraints. Achuta Kadambi et al. provide an overview of various techniques to incorporate physics into data-driven vision pipelines.