Freenets: Learning Layerfree Neural Network Topologies

Yerram Varun, Rahul Madhavan

Keywords: NAS, Neural Architecture Search, Neural Architecture, Neural Coactivation Matrix, AutoML, Neural Connectivity Graph

TL;DR: We propose an approach to neural network architecture based on neural pruning and expansion based on a neural coactivation matrix

Abstract: We propose a novel data driven approach to neural architectures based on information flows in a Neural Connectivity Graph (NCG). This technique gives rise to a category of neural networks that we call Free Networks, characterized entirely by the edges of an acyclic uni-directional graph. Further, we design a unique, data-informed methodology to systematically prune and augment connections in the proposed architecture during training. We show that any layered feed forward architecture is a subset of the class of Free Networks. Therefore, we propose that our method can produce a class of neural graphs that is a superset of any existing feed-forward networks. Additionally, we demonstrate the existence of certain classes of data, which are expressible through FreeNets, but not through any other feedforward architecture over the same number of neurons. We perform extensive experiments on this new architecture, to visualize the evolution of the neural topology over real world datasets, and showcase its performance alongside comparable baselines.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5367