FGPTQ-ViT: Fine-Grained Post-training Quantization for Vision Transformers

Caihua Liu, Hongyang Shi, Xinyu He

Abstract: The complex architecture and high training cost of Vision Transformers (ViTs) have prompted the exploration of post-training quantization (PTQ). However, introducing previous PTQ into ViTs performed worse because the activation values after processing by the softmax and GELU functions were extremely unbalanced distributed and not the common Gaussian distribution. To solve this problem, we propose a fine-grained ViT quantization method to fit this special distribution and reduce the quantization error of the activation values. We also design an adaptive piecewise point search algorithm that can automatically find the optimal piecewise point. Both the piecewise point and its search process are in the form of a power of two, making it possible to be implemented on general-purpose hardware with a simple shift operation. Experiments show that the quantization algorithm requires only 32 calibration images and achieves nearly lossless prediction accuracy in the classification task of ImageNet dataset. (The accuracy degradation for 8-bit quantization does not exceed 0.45%, and the average degradation is 0.17%).