Variance Pruning: Pruning Language Models via Temporal Neuron Variance

Berry Weinstein, Yonatan Belinkov

Keywords: Natural language processing, Transformers, Language Models, Pruning

Abstract: As language models become larger, different pruning methods have been proposed to reduce model size. However, the typical sparsity patterns that are formed by commonly pruning regimes do not fully exploit the properties of modern hardware devices on which these models are being trained and deployed. Most known unstructured, or even structured, pruning regimes usually introduce requirements for additional hardware components to make these sparsity patterns useful. Here we propose a simple pruning algorithm, based on variance analysis of output neurons that correspond to entire rows of weights. Our algorithm facilitates the construction of row-sparse matrices, allowing an extremely convenient way of exploiting this sparsity on existing hardware architectures. Empirical experiments with natural language understanding tasks show that our method leads to little to no accuracy degradation, and at times even better accuracy, using a 50\% sparse BERT\textsubscript{LARGE} model.

One-sentence Summary: We prune entire weight rows in NLP Transformer-based models with output neurons variance analysis.