Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compression

ICLR 2026 Conference Submission16763 Authors

Keywords: Model Compression, Joint Model Compression, Compression Order, Network Pruning, Network Quantization

TL;DR: Neural networks compressed by multiple methods perform better when weaker perturbations are applied first and stronger ones later.

Abstract:

What happens when multiple compression methods are combined—does the order in which they are applied matter?
Joint model compression has emerged as a powerful strategy to achieve higher efficiency by combining multiple methods such as pruning and quantization.
A central but underexplored factor in joint model compression is the compression order, or the sequence of different methods within the compression pipeline.
Most prior studies have either sidestepped the issue by assuming orthogonality between techniques, while a few have examined them only in highly constrained cases.
Consequently, the broader role of compression order in shaping model performance remains poorly understood.
In this paper, we address the overlooked problem of compression order and provide both theoretical and empirical analysis.
We formulate the problem of optimizing the compression order and introduce the Progressive Intensity Hypothesis, which states that weaker perturbations should precede stronger ones.
We provide theoretical guarantees showing that the relative benefit of one order increases with the underlying performance gap.
Extensive experiments on both language and vision models validate the hypothesis, and further show its generality to broader setups such as multi-stage compression and mixed-precision quantization.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 16763