Expressivity of Shallow Neural Networks Over Finite Fields

ICLR 2026 Conference Submission22921 Authors

Keywords: Polynomial neural networks, finite fields, expressivity, neuromanifold

Abstract: We study the expressivity of shallow polynomial neural networks (PNNs) with monomial activation functions over finite fields. For a given architecture, we define a neuromanifold as the image of the map from all possible network weights into the product of polynomial rings. We quantify the expressivity by the cardinality of the neuromanifold, and derive a natural lower and upper bound. This leads to counting rational points over finite fields, a problem closely linked to the Weil conjectures. Finally, we present an architecture that exhibits a striking difference in the neuromanifolds when considered over a characteristic zero versus a finite‐characteristic field, illustrating the critical role of field characteristic on the notion of expressivity.

Primary Area: learning theory

Submission Number: 22921