ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning

ICLR 2025 Conference Submission1125 Authors

Keywords: PINNs, SciML, PDEs, Natural Gradient, Neural Tangent Kernel

TL;DR: We propose a new natural gradient that scales as $\min(P^2S, S^2P)$, $P$ the number of parameters and $S$ the batch size, focusing on its application to PINNs traning.

Abstract:

In the recent years, Physics Informed Neural Networks (PINNs) have received strong interest as a method to solve PDE driven systems, in particular for data assimilation purpose. This method is still in its infancy, with many shortcomings and failures that remain not properly understood.
In this paper we propose a natural gradient approach to PINNs which contributes to speed-up and improve the accuracy of the training.
Based on an in depth analysis of the differential geometric structures of the problem, we come up with two distinct contributions:
(i) a new natural gradient algorithm that scales as $\min(P^2S, S^2P)$, where $P$ is the number of parameters, and $S$ the batch size;
(ii) a mathematically principled reformulation of the PINNs problem that allows the extension of natural gradient to it, with proved connections to Green's function theory.

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1125