SlimLLM: Accurate Structured Pruning for Large Language Models

Jialong Guo, Xinghao Chen, Yehui Tang, Yunhe Wang

Abstract: Large language models(LLMs) have garnered significant attention and demonstrated impressive capabilities in a wide range of applications. However, due to their enormous computational costs, the deployment and application of LLMs are often severely limited. To address this issue, structured pruning is an effective solution to compress the parameters of LLMs. Determining the importance of each sub-module in LLMs and minimizing performance loss are critical issues that need to be carefully addressed in structured pruning. In this paper, we propose an effective and fast structured pruning method named SlimLLM for large language models. For channel and attention head pruning, we evaluate the importance based on the entire channel or head, rather than merely aggregating the importance of individual elements within a sub-module. This approach enables a more holistic consideration of the interdependence among elements within the sub-module. In addition, we design a simple linear regression strategy for the output matrix to quickly recover performance. We also propose layer-based importance ratio to determine the pruning ratio for each layer. Based on the LLaMA benchmark results, our SlimLLM outperforms other methods and achieves state-of-the-art performance.

Lay Summary: Large Language Models (LLMs) represent a highly promising computing paradigm, but deployment is constrained by substantial computational requirements. We focus on pruning and compensating model parameters based on their output relevance. This approach helps minimize performance degradation while enhancing inference efficiency.

Primary Area: Deep Learning->Large Language Models

Keywords: structure pruning, large language models, machine learning

Submission Number: 14414