LoD: Unlocking Performance Gains in Compression via Differential Analysis

Boyang Zhang, Daning Cheng, Yunquan Zhang, Fangming Liu

Keywords: Mixed-precision; Lossless;

TL;DR: Mixed-precision compression lead to performance improvement

Abstract: Mixed-precision compression reduces model size and enhances inference efficiency, yet mainstream research focuses on minimizing accuracy loss from parameter compression. However, experimental evidence and observations often reveals performance improvements under specific conditions, challenging the assumed performance-efficiency tradeoff. These gains, often attributed to fortuitous alignments, lack systematic explanation and exhibit instability in performance across models and datasets. This work investigates these phenomena using a loss-driven framework based on total differential analysis, addressing three interconnected questions: \textcolor{magenta}{\textbf{(1)}} What conditions enable mixed-precision compression to enhance performance? \textcolor{magenta}{\textbf{(2)}} How can we model and control performance instability to ensure lossless outcomes? \textcolor{magenta}{\textbf{(3)}} What are the theoretical boundaries for achieving lossless compression? We take two mainstream compression methods as examples, parameter decomposition and quantization and propose a loss-driven(LoD) theoretical framework. For decomposition, we optimize layer-wise ranks within lossless neighborhoods. For quantization, we formulate compression as a grouped knapsack optimization problem. Extensive experiments across diverse datasets and architectures validate consistent, stable gains. And the code will be released.

Primary Area: interpretability and explainable AI

Submission Number: 8254