TESLA: Taylor Expansion of Sinusoidal Learnable Activations

Daehwa Ko, JaeHyeon Kim, SeungHyun Ham, Jay Hoon Jung

TL;DR: harmonic (\ell_1) budget that controls polynomial order, stabilizes training via Lipschitz bounds, supports Rademacher/NTK analyses, and improves high-order tasks and ImageNet-100.

Abstract: The parity problem—deciding whether the number of ones in a binary vector is odd or even—remains challenging for standard neural networks due to linear inseparability and the need for global interactions. We propose TESLA, an activation defined as a learnable combination of sine and cosine terms, enabling explicit control over polynomial degree and selective amplification of high-order components. Theoretically, we show that constraining TESLA’s coefficients yields Lipschitz/Rademacher complexity bounds and shapes the training dynamics to emphasize higher-frequency structure. Empirically, on parity with input length $n=32$, TESLA attains strong generalization with 100K training samples ($\approx 0.002\%$ of the $2^{32}$ input space) and remains robust under heavy corruption, retaining high accuracy with up to 30\% label noise. We also compare against periodic and frequency-based baselines (SIREN, SNAKE, and Fourier feature embeddings) on parity and Forrelation. Beyond synthetic structure, TESLA delivers comparable performance on ImageNet-100, indicating that activation-level degree control transfers to more general vision workloads. Code: \url{https://github.com/KAU-QuantumAILab/TESLA}

Code Dataset Promise: Yes

Code Dataset Url: https://github.com/KAU-QuantumAILab/TESLA

Format Confirmation: I agree that I have read and followed the formatting instructions for the camera ready version.

Submission Number: 2415