Provably high degree polynomial representation learning with neural networks

Rahul Madhavan, Yerram Varun

Keywords: neural networks, polynomial representations

TL;DR: We can obtain n many monomials of degree n with a neural network consisting of n neurons

Abstract:

We propose a novel data driven approach to neural architectures based on information flows in a Neural Connectivity Graph (NCG). This technique gives rise to a category of neural networks that we call ``Free Networks'', characterized entirely by the edges of an acyclic uni-directional graph. Furthermore, we design a unique, data-informed methodology to systematically prune and augment connections in the proposed architecture during training. We show that any layered feed forward architecture is a subset of the class of Free Networks. Therefore, we propose that our method can produce a class of neural graphs that is a superset of any existing feed-forward networks. Our analysis provably guarantees that FreeNets with $k$ neurons can exactly represent any polynomial of degree $k$.
We perform extensive experiments on this new architecture, to visualize the evolution of the neural topology over real world datasets, and showcase its performance alongside comparable baselines.

Submission Number: 16