SONIC: Spectral Oriented Neural Invariant Convolutions

ICLR 2026 Conference Submission17090 Authors

Keywords: Spectral Neural Networks, Spectral Parameterization, Resolution Invariance, State-Space Models, Spectral Factorization, Convolution Alternatives, Oriented Filters, Global Receptive Fields, Robust Representation Learning

TL;DR: SONIC filters combine global receptive fields with directionality, interpretability, and resolution invariance, achieved in a highly parameter-efficient way

Abstract: Convolutional Neural Networks (CNNs) rely on fixed-size kernels scanning local patches, which limits their ability to capture global context or long-range dependencies without very deep architectures. Vision Transformers (ViTs), in turn, provide global connectivity but lack spatial inductive bias, depend on explicit positional encodings, and remain tied to the initial patch size. Bridging these limitations requires a representation that is both structured and global. We introduce SONIC (Spectral Oriented Neural Invariant Convolutions), a compact collection of spectral filters that learns directly in the Fourier domain. SONIC factorises multi-channel frequency responses through a small set of shared oriented components. This yields filters that are directional, interpretable, and resolution-invariant, extending globally beyond patch-size limitations. Parameters scale linearly with the number of channels, enabling efficient learning without loss of expressivity. Experiments on standard vision benchmarks show that SONIC delivers more robust performance than conventional models, while matching or exceeding their accuracy with substantially fewer parameters.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 17090