VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers

Xianghong Fang, Yuan Yuan, Dehan Kong, Tim G. J. Rudner

Keywords: VQ-Transplant, Plug-and-play integration, Computational cost reduction, Pre-trained tokenizers

TL;DR: VQ-Transplant enables efficient plug-and-play integration of new vector quantization techniques into frozen models with minimal computational overhead, achieving high fidelity while reducing training costs by 95%.

Abstract: Vector Quantization (VQ) underpins modern discrete visual tokenization. However, training quantization modules for state-of-the-art VQ-based models requires significant computational resources which, in practice, all but prevents the development of novel, cutting-edge VQ techniques under resource constraints. To address this limitation, we propose VQ-Transplant, a simple framework that enables plug-and-play integration of new VQ modules into frozen, pre-trained tokenizers by replacing their native VQ modules. Crucially, the proposed transplantation process preserves all encoder-decoder parameters, obviating the need for costly end-to-end retraining when modifying the quantization method. To mitigate decoder-quantization mismatch, we introduce a lightweight decoder adaptation strategy (trained for only 5 epochs on ImageNet-1k) to align feature priors with the new quantization space. In our empirical evaluation, we find that VQ-Transplant allows obtaining near state-of-the-art reconstruction fidelity for industry-level models like VAR while reducing the training cost by 95%. VQ-Transplant democratizes quantization research by enabling resource-efficient integration of novel VQ techniques while matching industry-level reconstruction performance.

Primary Area: generative models

Submission Number: 2487