Soon Filter: Advancing Feed-Forward Neural Architectures for Inference at the Edge

Alan Tendler Leibel Bacellar, Zachary Susskind, Mauricio Breternitz Jr, Lizy Kurian John, Felipe M.G. França, Priscila Machado Vieira Lima

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Binary Neural Networks, DeepShift, ULEEN, Weightless Neural Networks

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: As Deep Neural Networks become more complex and computationally demanding, efficient models for inference at the edge, particularly multiplication-free ones, have gained significant attention. The Ultra Low-Energy Edge Neural Network (ULEEN) is a notable architecture optimized for feed-forward designs. ULEEN uniquely employs Bloom Filters with binary values to compute neuron activation, boasting better efficiency metrics than Binary Neural Networks (BNNs). This work uncovers a gradient back-propagation bottleneck within ULEEN's Bloom filters and introduces introduces a simplified version of it as a solution: the "Soon Filter". Both theoretically and empirically, we demonstrate that our approach improves gradient back-propagation efficiency. Tests on various UCI datasets and MNIST, which are standard benchmarks for feed-forward models, reveal that our method surpasses ULEEN, BNN, and DeepShift. Notably, with MNIST, we achieve 98.6% with only 98KiB, while ULEEN, BNN and DeepShift achieves 98.5% with 262KiB, 98.5% with 355KiB and 98.3% with 408KiB respectively. Furthermore, when using MLPerf Tiny datasets, which are typically more appropriate for CNNs, we consistently outperform other models when they are implemented as Multilayer Perceptrons. This results underscores the promising potential of our solution for efficient inference at the edge in applications that rely on feed-forward architectures.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3884