ON TRAINING DERIVATIVE-CONSTRAINED NEURAL NETWORKS
Keywords: Scientific Machine learning, Physics-informed neural networks, Derivative-constrained
TL;DR: We propose methods to improve training of derivative-constrained NNs commonly found in physics-based applications.
Abstract: We refer to the setting where the (partial) derivatives of a neural network’s (NN’s)
predictions with respect to its inputs are used as additional training signal as a
derivative-constrained (DC) NN. This situation is common in physics-informed
settings in the natural sciences. We propose an integrated RELU (IReLU) acti-
vation function to improve training of DC NNs. We also investigate denormal-
ization and label rescaling to help stabilize DC training. We evaluate our meth-
ods on physics-informed settings including quantum chemistry and Scientific Ma-
chine Learning (SciML) tasks. We demonstrate that existing architectures with
activations replaced with IReLU activations combined with denormalization/label
rescaling better incorporate training signal provided by derivative constraints.
Supplementary Material: zip
Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6220
Loading