CompSRT: Quantization and Pruning for Image Super Resolution Transformers

Dorsa Zeinali, Hailing Wang, Yitian Zhang, Yun Fu

Keywords: Compression, Quantization, Pruning, Image Super Resolution, Transformer, Swin-IR

Abstract: Model compression has emerged as a way to reduce the cost of using image super resolution models by decreasing storage size and inference time. However, the gap between the best compressed models and the full precision model still remains large and a need for deeper understanding of compression theory on more performant models remains. Prior research on quantization of LLMs has shown that Hadamard transformations lead to weights and activations with reduced outlier, which leads to improved performance. We argue that while the Hadamard transform does reduce the effect of outliers, an empirical analysis on how the transform functions remains needed. By studying the distributions of weights and activations of SwinIR-light, we show with statistical analysis that lower errors is caused by the Hadamard transforms ability to reduce the ranges, and increase the proportion of values around $0$. Based on these findings, we introduce CompSRT, a more performant way to compress the image super resolution transformer network SwinIR-light. We perform Hadamard-based quantization, and we also perform scalar decomposition to introduce two additional trainable parameters. Our quantization performance statistically significantly surpasses the SOTA in metrics with gains as large as 1.53 dB, and visibly improves visual quality by reducing blurriness at all bitwidths. At 3-4 bits, to show our method is compatible with pruning for increased compression, we also prune 40% of weights and show that we can achieve 6.67-15% reduction in bits per parameter with comparable performance to SOTA.

Primary Area: optimization

Submission Number: 20317