A Unifying View of Vector, Product and Scalar Quantization: An Information-Theoretic Perspective

Xianghong Fang, Wenlong Mou, Yuan Yuan, Dehan Kong, Tim G. J. Rudner

Keywords: Unified View, Vector Quantization, Product Quantization, Scalar Quantization, Information-Theory

TL;DR: We present a unified view of VQ, PQ, and SQ by taking an information theoretic perspective.

Abstract: Discrete visual tokenization, predominantly driven by vector, scalar, and product quantization, lacks a unifying conceptual framework that elucidates the impact and tradeoffs of different quantization optimization objectives. In this paper, we propose a unified information-theoretic framework to shed light on these considerations. To do so, we view quantization as information compression and define the information loss (quantization error), compression ratio, and input/output as information-theoretic quantities. Using this framework, we resolve three central open questions: First, we theoretically prove and empirically demonstrate that minimizing quantization error, rather than maximizing codebook utilization, is the paramount optimization objective for ensuring training stability and reconstruction fidelity. Second, we establish two critical fairness conditions for intrinsic algorithm comparison: controlling the latent feature distribution variance and ensuring identical compression ratios. Third, we demonstrate, both theoretically and empirically, that under these conditions, modern vector quantization outperforms scalar and product quantization at minimizing quantization error. Our work provides a foundational reframing of quantization algorithms, resolving conceptual ambiguities and providing the first artifact-free comparison that establishes quantization error minimization as the core optimization criterion.

Primary Area: generative models

Submission Number: 2688