EGGS-PTP: An Expander-Graph Guided Structured Post-training Pruning Method for Large Language Models

Omar Bazarbachi, Zijun Sun, Yanning Shen

Keywords: Large Language Models, Structured Sparsity, Expander Graphs

TL;DR: A novel structured pruning method for Large language Models is proposed by incorporating expander graphs, which reduces the computation and memory burdens while preserving performance.

Abstract: As Large Language Models (LLMs) become more widely adopted and scale up in size, the computational and memory challenges involved in deploying these massive foundation models have grown increasingly severe. This underscores the urgent need to develop more efficient model variants. Faced with this challenge, the present work introduces EGGS-PTP: an Expander-Graph Guided Structured Post-training Pruning method. The proposed approach leverages graph theory to guide the design of N:M structured pruning, effectively reducing model size and computational demands. By incorporating concepts from expander graphs, EGGS-PTP ensures information flow within the pruned network, preserving essential model functionality. Extensive numerical experiments demonstrate that EGGS-PTP not only achieves significant acceleration and memory savings due to structured sparsity but also outperforms existing structured pruning techniques in terms of accuracy across various LLMs.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 13511