HPC: Hierarchical Progressive Coding Framework for Volumetric Video

Zihan Zheng, Houqiang Zhong, Qiang Hu, Xiaoyun Zhang, Li Song, Ya Zhang, Yanfeng Wang

Abstract: Volumetric video based on Neural Radiance Field (NeRF) holds vast potential for various 3D applications, but its substantial data volume poses significant challenges for compression and transmission. Current NeRF compression lacks the flexibility to adjust video quality and bitrate within a single model for various network and device capacities. To address these issues, we propose HPC, a novel hierarchical progressive volumetric video coding framework achieving variable bitrate using a single model. Specifically, HPC introduces a hierarchical representation with a multi-resolution residual radiance field to reduce temporal redundancy in long-duration sequences while simultaneously generating various levels of detail. Then, we propose an end-to-end progressive learning approach with a multi-rate-distortion loss function to jointly optimize both hierarchical representation and compression. Our HPC trained only once can realize multiple compression levels, while the current methods need to train multiple fixed-bitrate models for different rate-distortion (RD) tradeoffs. Extensive experiments demonstrate that HPC achieves flexible quality levels with variable bitrate by a single model and exhibits competitive RD performance, even outperforming fixed-bitrate models across various datasets.

Primary Subject Area: [Systems] Transport and Delivery

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: Our work contributes to the field of multimedia/multimodal processing by tackling the pressing need for efficient representation, coding, and streaming transmission of 3D volumetric video, which has sparked widespread interest and is very important in modern multimedia due to its immersive experience in VR, AR, and telepresence. However, the huge data requirements of such videos present a significant difficulty for transmission, especially over limited bandwidth or on devices with varying capabilities. Addressing these challenges, our study introduces HPC, a novel hierarchical progressive coding framework for volumetric video. Our solution prioritizes the streamability of volumetric content, ensuring that it can be delivered efficiently across diverse network conditions without re-compromising on quality. By doing so, this work bridges the gap between the high data demands of 3D volumetric videos and the practical limitations of current transmission and rendering technologies, marking a step forward in making high-quality, immersive experiences more accessible to users regardless of their network conditions or device capabilities. Overall, our method represents the pioneering solution for progressive volumetric video coding, poised to significantly contribute to advancements in modern multimedia.

Submission Number: 2585