Reduce What You Use: Input‑Aware Matrix‑Multiplication Pruning for LLMs

ICLR 2026 Conference Submission23153 Authors

Keywords: Redundancy, Matrix pruning, Training-free, Large Language Models

TL;DR: We consider LLM as a system with observable redundancy and propose a post-training, input-adaptive, and general matrix multiplication pruning algorithm.

Abstract: Transformer-based language models achieve strong performance but at high computational cost, raising the question of whether their full dimensional capacity is necessary at inference. We introduce Reduced Matrix-Multiplication (RMM), a training-free rule that adaptively prunes feature dimensions on the fly. Given current activations, RMM scores hidden channels with simple norms, retains a controlled fraction, and performs multiplications only within this reduced subspace—yielding deterministic approximations without altering model weights. Applied uniformly across all linear operations, RMM exposes a smooth accuracy–efficiency frontier governed by a single retention ratio. Across models ranging from 1B to 70B parameters and tasks spanning question answering, reasoning, math, coding, summarization, and vision–language benchmarks, RMM achieves substantial cost reductions with minimal accuracy loss. Larger models tolerate more aggressive pruning, highlighting increasing representational redundancy at scale. These findings demonstrate that high-dimensional computations in LLMs can be systematically compressed, offering a simple and general mechanism for controllable accuracy–efficiency tradeoffs

Primary Area: generative models

Submission Number: 23153