MCNC: Manifold-Constrained Reparameterization for Neural Compression

Chayne Thrash, Reed Andreas, Ali Abbasi, Parsa Nooralinejad, Soroush Abbasi Koohpayegani, Hamed Pirsiavash, Soheil Kolouri

Keywords: Model Compression, LoRA, PEFT, Transformers, ViT

TL;DR: A method that compresses large models by limiting parameters to low-dimensional manifolds. It achieves superior compression of CNNs, ViTs, and LLMs across tasks compared to current techniques.

Abstract:

The outstanding performance of large foundational models across diverse tasks,
from computer vision to speech and natural language processing, has significantly
increased their demand. However, storing and transmitting these models poses
significant challenges due to their massive size (e.g., 750GB for Llama 3.1 405B).
Recent literature has focused on compressing the original weights or reducing the
number of parameters required for fine-tuning these models. These compression
methods generally constrain the parameter space, for example, through low-rank
reparametrization (e.g., LoRA), pruning, or quantization (e.g., QLoRA) during
or after the model training. In this paper, we present a novel model compres-
sion method, which we term Manifold-Constrained Neural Compression (MCNC).
This method constrains the parameter space to low-dimensional pre-defined and
frozen nonlinear manifolds, which effectively cover this space. Given the preva-
lence of good solutions in over-parameterized deep neural networks, we show that
by constraining the parameter space to our proposed manifold, we can identify
high-quality solutions while achieving unprecedented compression rates across
a wide variety of tasks and architectures. Through extensive experiments in
computer vision and natural language processing tasks, we demonstrate that our
method significantly outperforms state-of-the-art baselines in terms of compres-
sion, accuracy, and/or model reconstruction time. Our code is publicly available at
https://github.com/mint-vu/MCNC.

Primary Area: other topics in machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8392