Learning Input Encodings for Kernel-Optimal Implicit Neural Representations

Zhemin Li, Liyuan Ma, Hongxia Wang, Yaoyun Zeng, Xiaolong Han

Abstract: Implicit Neural Representations (INRs) rely heavily on architectural choices for good generalization. Developing theoretically grounded approaches for architecture design remains an active area of research. Via theoretical analysis of the infinite-width limit, we establish a methodology that characterizes INR's generalization by means of kernel alignment. We first formulate the optimal kernel that minimizes pointwise expected squared error, then demonstrate that the Neural Tangent Kernel of the composed function (INR with input encoding) can approximate any positive semidefinite dot-product kernels through input feature mapping adjustments. Building upon these insights, we propose a Kernel Alignment Regularizer (KAR) that naturally integrates with existing INR systems to enhance kernel alignment. We further develop Plug-in Encoding for Aligned Kernels (PEAK) to refine INR models with KAR using learnable input encoding. This work contributes to the ongoing research efforts in bridging theory and practice for principled INR architecture design. Code is available at https://github.com/lizhemin15/KAR.

Lay Summary:

Implicit Neural Representations (INRs) need careful architecture design to generalize well, but lack strong theoretical guidance. We analyze how INR generalization relates to kernel alignment, i.e., how well the model matches the optimal kernel. This leads us to create:
1. KAR: A regularizer that boosts kernel alignment in existing INRs.
2. PEAK: An algorithm to refine INR models with KAR using learnable input encoding.
By connecting theory with practice, our work improves INR's generalization.

Link To Code: https://github.com/lizhemin15/KAR

Primary Area: Deep Learning->Other Representation Learning

Keywords: INRs, Generalization, Encoding, Reconstruction

Submission Number: 3333