Neuroexplicit Diffusion Models for Inpainting of Optical Flow Fields

Tom Fischer, Pascal Peter, Joachim Weickert, Eddy Ilg

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Deep Learning, Diffusion, Inpainting, Optical Flow, Neuroexplicit

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: In this work we combine models based on Partial Differential Equations with a CNN into a hybrid model- and data-driven architecture for the task of inpainting optical flow fields.

Abstract:

Deep learning has revolutionized the field of computer vision by introducing large scale neural networks with millions of parameters.
Training these networks requires massive datasets and leads to intransparent models that can fail to generalize.
At the other extreme, models designed from partial differential equations (PDEs) embed specialized domain knowledge into mathematical equations and usually rely on few manually chosen hyperparameters.
This makes them transparent by construction and if designed and calibrated carefully, they can generalize well to unseen scenarios. In this paper, we show how to bring model- and data-driven approaches together by combining the explicit PDE-based approaches with convolutional neural networks to obtain the best of both worlds.
We illustrate a joint architecture for the task of inpainting optical flow fields and show that the combination of model- and data-driven modeling leads to an effective architecture.
Our model outperforms both fully explicit and fully data-driven baselines in terms of reconstruction quality, robustness and amount of required training data.
Averaging the endpoint error across different mask densities, our method outperforms the explicit baseline by $27.12$%, the GAN baseline by $46.72$% and the Probabilisitic Diffusion baseline by $42.38$%.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3351