No More DeLuLu: A Kernel-Based Activation-Free Neural Networks

Taha Bouhsine

Keywords: kernel learning; neural operators; universal approximation; geometric deep learning; information geometry; dynamical stability; physics-inspired models; interpretable representations

TL;DR: We propose the ⵟ-product, a new kernel-based neural operator that integrates nonlinearity and normalization directly into the core interaction.

Abstract: We introduce the ⵟ-product, a kernel operator that combines quadratic alignment with inverse-square interactions. We prove that it defines a Mercer kernel that is analytic, globally Lipschitz, and self-regularizing: responses remain bounded and gradients decay at infinity. Neural Matter Networks (NMNs), constructed as linear combinations of ⵟ-atoms, are universal approximators on compact domains without explicit nonlinear activations. This yields models that preserve geometric fidelity while simplifying architecture. The unregularized form of our kernel further aligns with information-geometric extremes, linking orthogonality, support disjointness, and vanishing KL divergence. Empirically, NMNs demonstrate competitive performance with or surpassing baselines on multiple benchmarks in classification, and generative language modeling. Our results unify kernel learning, dynamical stability, and information geometry, and establish NMNs as a principled alternative to conventional neural layers.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 5543