Neural Tangent Kernel Analysis and Filtering for Robust Fourier Feature Embedding

Mingze Ma, Qingtian Zhu, Yifan Zhan, Zhengwei Yin, Hongjun Wang, Jiancheng Zhao, Yinqiang Zheng

Keywords: Implicit Neural Representation, Spectral Bias, Positional Encoding, Fourier Features

Abstract: Implicit Neural Representations (INRs) employ neural networks to represent continuous functions by mapping coordinates to the corresponding values of the target function, with applications e.g., inverse graphics. However, INRs face a challenge known as spectral bias when dealing with scenes containing varying frequencies. To overcome spectral bias, the most common approach is the Fourier features-based methods such as positional encoding. However, Fourier features-based methods will introduce noise to output, which degrades their performances when applied to downstream tasks. In response, this paper addresses this problem by first investigating the underlying causes through the lens of the Neural Tangent Kernel. Through theoretical analysis, we propose that using Fourier features embedding can be interpreted as fitting Fourier series expansion of the target function, from which we find that it is the insufficiency in the finitely sampled frequencies that causes the generation of noisy outputs. Leveraging these insights, we introduce bias-free MLPs as an adaptive linear filter to locally suppress unnecessary frequencies while amplifying essential ones by adjusting the coefficients at the coordinate level. Additionally, we propose a line-search-based algorithm to adjust the filter's learning rate dynamically, achieving Pareto efficiency between the adaptive linear filter module and the INRs. Extensive experiments demonstrate that our proposed method consistently improves the performance of INRs on typical tasks, including image regression, 3D shape regression, and inverse graphics.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 223