Shortened LLaMA: A Simple Depth Pruning for Large Language Models
Keywords: Pruning, Structured Pruning, Large Language Models, Inference Efficiency
TL;DR: Compared to recent width pruning methods for LLMs, our depth pruning boosts inference speeds while yielding competitive zero-shot performance.
Abstract: Structured pruning of modern large language models (LLMs) has emerged as a way of decreasing their high computational needs. Width pruning reduces the size of projection weight matrices (e.g., by removing attention heads) while maintaining the number of layers. Depth pruning, in contrast, removes entire layers or blocks, while keeping the size of the remaining weights unchanged. Most current research focuses on either width-only or a blend of width and depth pruning, with little comparative analysis between the two units (width vs. depth) concerning their impact on LLM inference efficiency. In this work, we show that a simple depth pruning approach can compete with recent width pruning methods in terms of zero-shot task performance. Our pruning method boosts inference speeds, especially under memory-constrained conditions that require limited batch sizes for running LLMs, where width pruning is ineffective. We hope this work can help deploy LLMs on local and edge devices. Code and models can be found at: https://github.com/Nota-NetsPresso/shortened-llm
Submission Number: 21
Loading