A Unified Theory of Sinusoidal Activation Families for Implicit Neural Representations

Alireza Morsali, MohammadJavad Vaez, Mohammad Hossein Soltani, Amirhossein Kazerouni, Babak Taati, Morteza Mohammad-Noori

Keywords: Implicit Neural Representation, Sinusoidal Activations, Neural Tangent Kernel

TL;DR: We propose STAF, a trainable sinusoidal activation framework for INRs with learned amplitudes, frequencies, and phases. Theory links STAF to sine networks and NTK, while experiments show higher fidelity and faster convergence across different tasks.

Abstract:

Implicit Neural Representations (INRs) model continuous signals with compact neural networks and have become a standard tool in vision, graphics, and signal processing. A central challenge is accurately capturing fine detail without heavy hand-crafted encodings or brittle training heuristics. Across the literature, periodic activations have emerged as a compelling remedy: from SIREN, which uses a single sinusoid with a fixed global frequency, to more recent architectures employing multiple sinusoids and, in some cases, trainable frequencies and phases.
We study this *family* of sinusoidal activations and develop a principled theoretical and practical framework for trainable sinusoidal activations in INRs. Concretely, we instantiate this framework with **S**inusoidal **T**rainable **A**ctivation **F**unctions **(STAF)**, a Fourier-series activation whose amplitudes, frequencies, and phases are learned. Our analysis (i) establishes a Kronecker-equivalence construction that expresses trainable sinusoidal activations with standard sine networks and quantifies expressive growth, (ii) characterizes how the Neural Tangent Kernel (NTK) spectrum changes under trainable sinusoidal parameterization, and (iii) provides an initialization that yields unit-variance post-activations without asymptotic central limit theorem (CLT) arguments.
Empirically, on images, audio, shapes, inverse problems (super-resolution, denoising), and NeRF, STAF is competitive and often superior in reconstruction fidelity, with consistently faster early-phase optimization. While periodic activations can alleviate *practical manifestations* of spectral bias, our results indicate they do not eliminate it; instead, trainable sinusoids reshape the optimization landscape to improve the *capacity–convergence* trade-off.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 22709