NEPENTHE: Entropy-Based Pruning as a Neural Network Depth's Reducer

Zhu LIAO, Victor Quétu, Van-Tam Nguyen, Enzo Tartaglione

Keywords: Pruning, Compression, Entropy, Deep Learning

Abstract:

While deep neural networks are highly effective at solving complex tasks, their computational demands can hinder their usefulness in real-time applications and with limited-resources systems. Besides, it is a known fact that, for many downstream tasks, off-the-shelf models are over-parametrized. While classical structured pruning can reduce the network's width, the computation's critical path, namely the maximum number of layers encountered at forward propagation, apparently can not be reduced.

In this paper, we aim to reduce the depth of over-parametrized deep neural networks: we propose an e**N**tropy-bas**E**d **P**runing as a n**E**ural **N**etwork dep**TH**'s r**E**ducer (NEPENTHE) to alleviate deep neural networks' computational burden.
Based on our theoretical finding, NEPENTHE leverages "unstructured'' pruning to bias sparsity enhancement in layers with low entropy to remove them entirely. We validate our approach on popular architectures such as MobileNet, Swin-T and RoBERTa, showing that, when in the overparametrization regime, some layers are linearizable (hence reducing the model's depth) with little to no performance loss. The code will be publicly available upon acceptance of the article.

Primary Area: other topics in machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6350