Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

Artur Zagitov, Alexander Miasnikov, Maxim Krutikov, Artem Tsedenov, Vladimir Aletov, Gleb Molodtsov, Nail Bashirov, Aleksandr Beznosikov

Keywords: Tensor Decompositions, Large Language Models, Tucker Decomposition, Tensor Train Decomposition

Abstract: Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints. Tensor decompositions have emerged as a promising direction, offering compact parameterizations well suited to Transformer weight structures. However, existing studies evaluate these methods in narrow settings, leaving unclear whether tensorization is effective at large-scale deployment. We systematically evaluate tensor compression across dense and MoE architectures, establishing performance trade-offs grounded in both empirical analysis and theoretical derivation. We identify a fundamental mismatch between the shared subspaces assumed by tensor decompositions and the heterogeneous representations learned by modern LLMs, thereby delineating their practical limits and clarifying their viable role in large-scale deployment.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 60