Generative Latent Video Compression

Zongyu Guo, Zhaoyang Jia, Jiahao Li, Xiaoyi Zhang, Bin Li, Yan Lu

Keywords: Neural Video Compression, Perceptual Video Compression, Generative Models

TL;DR: We introduce Generative Latent Video Compression (GLVC) with a redesigned codec for compression in continuous latent space and achieves state-of-the-art perceptual quality with strong temporal consistency.

Abstract: Perceptual optimization is widely recognized as essential for neural compression, yet balancing the rate–distortion–perception tradeoff remains challenging. This difficulty is especially pronounced in video compression, where frame-wise quality fluctuations often cause perceptually optimized neural video codecs to suffer from flickering artifacts. In this paper, inspired by the success of latent generative models, we present Generative Latent Video Compression (GLVC), an effective framework for perceptual video compression. GLVC employs a pretrained continuous tokenizer to project video frames into a perceptually aligned latent space, thereby offloading perceptual constraints from the rate–distortion optimization. We redesign the codec architecture explicitly for the latent domain, drawing on extensive insights from prior neural video codecs, and further equip it with innovations such as unified intra/inter coding and a recurrent memory mechanism. Experimental results across multiple benchmarks show that GLVC achieves state-of-the-art performance in terms of DISTS and LPIPS metrics. Notably, our user study confirms GLVC rivals the latest neural video codecs at nearly half their rate while maintaining stable temporal coherence, marking a step toward practical perceptual video compression.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 4373