Plug-and-Fold: Weight-Preserving Structured Compression for Large Language Models

ICLR 2026 Conference Submission17293 Authors

Keywords: Model Compression, Structured Compression, Large Language Models

Abstract: Large Language Models (LLMs) have achieved remarkable performance across a wide range of tasks, but their growing size poses significant challenges for deployment and efficiency. Among existing model compression methods, structured pruning has emerged as a popular approach for reducing model size. However, pruning removes structural components such as layers, heads, or channels, which can disrupt pre-trained weights and lead to fragile recovery fine-tuning process. In this work, we propose Plug-and-Fold (PnF), a weight-preserving yet structurally effective compression method. Rather than removing weights or modifying the model architecture, PnF introduces lightweight, learnable adapter modules into the projection layers of attention and feed-forward networks. These adapters are trained while keeping the original weights frozen, and are later folded into the base weights via simple matrix multiplications. This process yields a compressed model that is structurally identical to the original and incurs no additional runtime overhead. We evaluate PnF across a variety of benchmarks and model scales, demonstrating consistent improvements over recent state-of-the-art structured compression baselines. Our results highlight that preserving the integrity of pretrained weights not only simplifies the compression pipeline, but also improves generalization and performance recovery in compressed LLMs.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 17293