Revisiting Large Language Model Pruning using Neuron Semantic Attribution

Yizhuo Ding, Guosheng Hu, Xinwei Sun, Yanwei Fu

Keywords: LLM pruning

TL;DR: We present a large-scale study of LLM pruning, introduce NSA—the first interpretability framework for pruned LMs that links neuron-level activation shifts to task-relevant semantics.

Abstract: Pruning large language models (LLMs) is an effective way to reduce computation while maintaining strong performance. While some studies have explored how pruning affects different tasks, most remain narrow in scope and overlook interpretability-driven analysis. In this work, we propose Neuron Semantic Attribution (NSA), a novel method that analyzes pruning through neuron-level semantics. NSA links neurons to task-relevant concepts, providing a fine-grained understanding of how pruning impacts model behavior and causes task-specific sensitivity. We also conduct a comprehensive empirical study across 24 tasks spanning diverse domains, examining the effects of pruning configurations, calibration data, and sparsity levels. Our findings demonstrate that NSA serves as a reliable tool for interpreting and guiding pruning, helping to bridge the gap between model compression and interpretability.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 7555