Cut Less, Fold More: Model Compression through the Lens of Projection Geometry

ICLR 2026 Conference Submission18352 Authors

Keywords: model compression, structured pruning, model folding, projection geometry

TL;DR: Folding - low-rank, geometry-aware projection - typically outperforms structured pruning for calibration-free compression, supported by a one-rank slack guarantee and results on >1000 CNN/ViT checkpoints

Abstract: Compressing neural networks without retraining is vital for deployment at scale. We study calibration-free compression through the lens of projection geometry: structured pruning is an axis-aligned projection, whereas model folding performs a low-rank projection via weight clustering. We formalize both as orthogonal operators and show that, within a rank distance of one, folding provably yields smaller parameter reconstruction error, and under mild smoothness assumptions, smaller functional perturbations than pruning. At scale, we evaluate >1'000 checkpoints spanning ResNet18, PreActResNet18, ViT-B/32, and CLIP ViT-B/32 on CIFAR-10 and ImageNet-1K, covering diverse training hyperparameters (optimizers, learning rates, augmentations, regularization, sharpness-aware training). We show that folding typically achieves higher post-compression accuracy, with the largest gains at moderate–high compression. The gap narrows and occasionally reverses at specific training setups. Our results position folding as a geometry-aware, calibration-free alternative to pruning that is often superior in practice and principled in theory.

Primary Area: optimization

Submission Number: 18352