Constructing Sparse Neural Architecture with Deterministic Ramanujan Graphs

Arindam Biswas, Suryam Arnav Kalra, Pabitra Mitra, BISWAJIT BASU

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Sparse neural networks, expander graphs, pruning, Ramanujan graphs

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A sparse neural architecture based on constructing deterministic Ramanujan graphs

Abstract: We present a sparsely connected neural network architecture constructed using the theory of Ramanujan graphs which provide comparable performance to a dense network. The method can be considered as a before-training, deterministic, weight free, pruning at initialization (PaI) technique. The deterministic Ramanujan graphs occur either as Cayley graphs of certain algebraic groups or as Ramanujan $r$-coverings of the full $(k,l)$ bi-regular bipartite graph on $k + l$ vertices. Sparse networks are constructed for bipartite graphs representing both the convolution and the fully connected layers. We experimentally show that the proposed sparse architecture provides comparable accuracy with a lower sparsity ratio than those achieved by previous approaches based on non-deterministic methods for benchmark datasets. In addition, they retain other desirable properties such as path connectivity and symmetricity.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7198