$\rm SP^3$: Enhancing Structured Pruning via PCA Projection

Yuxuan Hu, Jing Zhang, Zhe Zhao, Chen Zhao, Xiaodong Chen, Cuiping Li, Hong Chen

Abstract:

Structured pruning is a widely used technique for reducing the size of pre-trained language models (PLMs), but current methods often overlook the potential of compressing the hidden dimension (sp3) in PLMs, a dimension critical to model size and efficiency. This paper introduces a novel structured pruning approach, Structured Pruning with PCA Projection (sp3), targeting the effective reduction of
by projecting features into a space defined by principal components before masking.
Extensive experiments on benchmarks (GLUE and SQuAD) show that can reduce by 70\%, compress 94\% of the model, and maintain over 96\% accuracy, and outperform other methods that compress by 6\% in accuracy at the same compression ratio. has also proven effective with other models, including OPT and Llama. Our data and code are available at https://anonymous.4open.science/r/DA-WID-206E/README.md.