Approximating Latent Manifolds in Neural Networks via Vanishing Ideals

Nico Pelleriti, Max Zimmer, Elias Samuel Wirth, Sebastian Pokutta

TL;DR: We replace the deep network tail with a compact polynomial layer that’s fast, accurate, and often more generalizable.

Abstract: Deep neural networks have reshaped modern machine learning by learning powerful latent representations that often align with the manifold hypothesis: high-dimensional data lie on lower-dimensional manifolds. In this paper, we establish a connection between manifold learning and computational algebra by demonstrating how vanishing ideals can characterize the latent manifolds of deep networks. To that end, we propose a new neural architecture that (i) truncates a pretrained network at an intermediate layer, (ii) approximates each class manifold via polynomial generators of the vanishing ideal, and (iii) transforms the resulting latent space into linearly separable features through a single polynomial layer. The resulting models have significantly fewer layers than their pretrained baselines, while maintaining comparable accuracy, achieving higher throughput, and utilizing fewer parameters. Furthermore, drawing on spectral complexity analysis, we derive sharper theoretical guarantees for generalization, showing that our approach can in principle offer tighter bounds than standard deep networks. Numerical experiments confirm the effectiveness and efficiency of the proposed approach.

Lay Summary:

Modern AI systems are incredibly powerful, but they often require very large, deep networks with millions of parameters. This makes them expensive to run, hard to interpret, and slower to use in practice.

We explored whether we could keep the smart part of an AI system — its early ability to understand patterns — and replace the later, heavier parts with a simpler mathematical model. Specifically, we used a type of math called polynomial equations to describe the structure of the data the AI sees. By doing this, we built a much smaller, faster system that still performs just as well.

Our method cuts down the size of the network significantly, reduces the amount of computation, and offers better theoretical understanding of how and why the system makes decisions. This could make future AI models more efficient, easier to explain, and more accessible to use in real-world applications.

Link To Code: https://github.com/ZIB-IOL/approximating-neural-network-manifolds

Primary Area: Deep Learning->Other Representation Learning

Keywords: neural network, manifold hypothesis, vanishing ideal, polynomials, latent space, computational algebra

Submission Number: 4573