LLM-Guider: A Language-Guided Discovery of Symbolic Pruning Metrics for Post-Training Sparsity in LLMs

ACL ARR 2025 May Submission2150 Authors

Abstract: Large Language Models (LLMs) have achieved remarkable advancements in natural language understanding, yet their mammoth size coupled with substantial training and inference costs can make them difficult to use in environments with limited resources. To address both memory and efficiency concerns, post-training unstructured sparsity techniques have emerged focusing on developing optimal pruning criteria to eliminate redundant weights while maintaining performance. However, these approaches often rely on manually crafted pruning criteria, leading to sub-optimal solutions due to heuristic oversimplifications. Therefore, we introduce LLM-Guider, a language-guided symbolic formula optimization framework that seeks to discover optimal pruning criteria through a transparent and systematic process. LLM-Guider comprises three interrelated stages: example selection, formula generation, and formula evaluation, which collectively enable the efficient exploration of the formula space. In addition, LLM-Guider enables incorporation of intuition, domain and mathematical knowledge through role prompts, hints and in-context examples. We also extend the standard set of aggregation strategies over calibration dataset, resulting in never-seen-before pruning metrics. Through extensive experiments, we demonstrate that formulas discovered through LLM-Guider is able to find formulas, which outpeform established baselines.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: Efficient/Low-Resource Methods for NLP

Contribution Types: NLP engineering experiment, Approaches to low-resource settings

Languages Studied: English

Submission Number: 2150