LLMGD: Compressing LLMs with Layerwise Geodesic Distances

ICLR 2026 Conference Submission20880 Authors

Keywords: Large Language Models, LLM, Model Compression, Probabilistic Graphical Models, Bi-Lipschitz Maps, Riemannian Manifold, Spectral Geometry

TL;DR: We introduce LLMGD, a spectral geodesic metric that quantifies layer-wise distortion in LLMs via bi-Lipschitz bounds, with applications in efficient model compression and robustness analysis.

Abstract: Understanding how internal representations evolve across layers in large language models (LLMs) is critical for interpretability, robustness, and efficient model design. We introduce LLMGD, a stability-guided pruning metric grounded in spectral graph theory. For each layer, we estimate a precision matrix from embedding vectors that characterize the model’s internal states, and then compute the geodesic distance on the cone of symmetric positive definite (SPD) matrices between successive layers. This yields a smooth and robust measure of representational distortion, identifying layers with minimal geodesic change as candidates for removal or replacement, thereby providing a principled foundation for model compression. Empirically, across multiple LLMs and tasks, including OPT-1.3B and OPT-2.7B models, LLMGD consistently detects structurally redundant segments and, when combined with lightweight replacement layers, delivers strong compression–accuracy trade-offs compared to existing pruning methods. We further establish a bi-Lipschitz upper-bound interpretation of LLMGD, which clarifies its robustness as a pruning criterion. Together, these results demonstrate that LLMGD reliably identifies structurally important layers and enables robust model compression with minimal performance degradation.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 20880