LLM-Guider: A Language-Guided Discovery of Symbolic Pruning Metrics for Post-Training Sparsity in LLMs

TMLR Paper7839 Authors

Abstract:

Large Language Models (LLMs) have achieved remarkable advancements in natural language understanding, yet their mammoth size, coupled with substantial training and inference costs, can make them difficult to use in environments with limited resources. To address both memory and efficiency concerns, post-training unstructured sparsity techniques have emerged, focusing on developing optimal pruning criteria to eliminate redundant weights while maintaining performance. However, these approaches often rely on manually crafted pruning criteria, leading to sub-optimal solutions due to heuristic oversimplifications. Therefore, we introduce LLM-Guider, a language-guided symbolic formula optimization framework that seeks to discover optimal pruning criteria through a transparent and systematic process. LLM-Guider comprises three interrelated stages: example selection, formula generation, and formula evaluation, which collectively enable the efficient exploration of the formula space. In addition, LLM-Guider enables the incorporation of intuition, domain, and mathematical knowledge through role prompts, hints, and in-context examples. We also extend the standard set of aggregation strategies over a calibration dataset, resulting in never-seen-before pruning metrics.
Through extensive experiments, we demonstrate that formulas discovered through LLM-Guider is able to find formulas that outperform established baselines.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Zhengzhang_Chen1

Submission Number: 7839