HybridFlow: Infusing Continuity into Masked Codebook for Extreme Low-Bitrate Image Compression

Lei Lu, Yanyue Xie, Wei Jiang, Wei Wang, Xue Lin, Yanzhi Wang

Abstract: This paper investigates the challenging problem of learned image compression (LIC) with extreme low bitrates. Previous LIC methods based on transmitting quantized continuous features often yield blurry and noisy reconstruction due to the severe quantization loss. While previous LIC methods based on learned codebooks that discretize visual space usually give poor-fidelity reconstruction due to the insufficient representation power of limited codewords in capturing faithful details. We propose a novel dual-stream framework, HyrbidFlow, which combines the continuous-feature-based and codebook-based streams to achieve both high perceptual quality and high fidelity under extreme low bitrates. The codebook-based stream benefits from the high-quality learned codebook priors to provide high quality and clarity in reconstructed images. The continuous feature stream targets at maintaining fidelity details. To achieve the ultra low bitrate, a masked token-based transformer is further proposed, where we only transmit a masked portion of codeword indices and recover the missing indices through token generation guided by information from the continuous feature stream. We also develop a bridging correction network to merge the two streams in pixel decoding for final image reconstruction, where the continuous stream features rectify biases of the codebook-based pixel decoder to impose reconstructed fidelity details. Experimental results demonstrate superior performance across several datasets under extremely low bitrates, compared with existing single-stream codebook-based or continuous-feature-based LIC methods.

Primary Subject Area: [Experience] Interactions and Quality of Experience

Secondary Subject Area: [Systems] Transport and Delivery

Relevance To Conference: Due to the explosive growth in demand for image transmission, significant challenges have emerged in terms of network bandwidth and information storage requirements. Thus, there is an urgent need for an efficient and effective image compression and transmission method at extremely low bitrates. Considering the ultra-low bit scenarios, previous single-stream structures have their unique advantages but suffer from inherent limitations that are difficult to overcome. Therefore, we propose a dual-stream approach called "HybridFlow," which combines continuous image feature-based LIC and discrete codebook-based LIC. Through effective and unique bridging design, "HybridFlow" integrates the advantages of single-stream structures, providing high-quality and high-fidelity resorted images while maintaining extremely low information transmission rates.

Submission Number: 2170