How Neural Networks With Derivative Labels Work: A Neural Tangent Kernel Perspective

Yuanqu Mou, zhenbin Wu, Chang Gou, Haiyang Bai, Jia Liu

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Application of Neural Tangent Kernel

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Deep neural networks have achieved impressive results in a range of fields, while their analytical properties have been slow to develop. Recently, a theoretical tool called Neural Tangent Kernel (NTK) has been proposed and extended to diverse architectures based on neural networks. This tool helps explain their convergence and generalization with a least-square loss. However, researchers in numerous fields have trained their networks using an additional derivative loss item, such as Jacobian Regularization and PINN (Physics-Informed Neural Networks). This loss setup has generality, while it often leads to challenging convergence issues. To address this problem, we propose a general paradigm that utilizes Gâteaux derivative labels to describe all these tasks. Additionally, we extend NTK in our setup from an analytical perspective and propose a geometrical perspective of parameter updating directions to explain the hard convergence. We also conduct experiments to verify our propositions. Finally, we provide specific expressions for distinct tasks within our paradigm.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3203