PATCH: Learnable Tile-level Hybrid Sparsity for LLMs

ICLR 2026 Conference Submission14633 Authors

Keywords: LLM Pruning, Hybrid Sparsity, LLM Compression, Semistructured Sparsity, 2:4 Sparsity

TL;DR: We propose PATCH, a learnable tile-level hybrid sparsity method for LLMs that adaptively mixes dense and 2:4 sparse tiles, achieving higher accuracy, hardware efficiency, and speedups.

Abstract: Large language models (LLMs) deliver impressive performance but incur prohibitive memory and compute costs at deployment. Model pruning is an effective way to reduce these overheads, yet existing approaches face challenges: unstructured sparsity, where nonzeros can appear anywhere, preserves accuracy but yields irregular access patterns that prevent GPU acceleration, while semi-structured 2:4 sparsity is hardware-friendly but enforces a rigid 50% pattern that degrades model quality. To bridge this gap, we introduce PATCH, a hybrid sparsity framework that enables a continuous sparsity ratio between 0% and 50%. PATCH partitions weight matrices into tiles, assigning each tile to be either dense or 2:4 sparse via a learnable mask selection mechanism. This design provides fine-grained control over accuracy–acceleration tradeoffs and supports non-uniform sparsity across layers, leading to superior overall quality. Across models from 0.5B to 8B parameters, PATCH consistently narrows the gap to dense accuracy while delivering practical speedups. For instance, on LLaMA-2 7B with an A6000 GPU, PATCH achieves 1.18×–1.38× end-to-end speedup over dense baselines while improving accuracy by 0.37%–2.96% compared to the state-of-the-art 2:4 pruning method, MaskLLM.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 14633