CSVQ: Channel-wise Shared-Codebook Vector Quantization for Stable and Expressive Discrete Representations

GangminPark, Hyuk-Yoon Kwon

Keywords: Vector Quantization, Discrete Representation Learning, Channel-wise Quantization; Shared Codebook

Abstract: Vector quantization (VQ) is a cornerstone of discrete representation learning, but existing methods often depend on very large codebooks that increase capacity while hurting stability and utilization. We present Channel-wise Shared-codebook Vector Quantization (CSVQ), a simple tokenizer that quantizes each latent channel independently using one shared scalar codebook with per-channel normalization. CSVQ achieves the same representational capacity as vector-quantized latents while using a much smaller codebook, reduces gradient noise through channel-wise aggregation, and cuts codebook memory to scale linearly with the number of codewords rather than with both codewords and channels. Across multiple datasets and settings, CSVQ improves reconstruction quality and training stability, remains competitive under strict memory budgets, and scales favorably with model capacity. Finally, CSVQ shows improvements in reconstruction quality and downstream task performance compared to the state-of-the-art VQ methods. Ablation and multi-seed studies support the design choices. Code is available at https://anonymous.4open.science/r/csvq-CF06.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 8174