ResVR: Joint Rescaling and Viewport Rendering of Omnidirectional Images

Weiqi Li, Shijie Zhao, Bin Chen, Xinhua Cheng, Junlin Li, Li zhang, Jian Zhang

Abstract: With the advent of virtual reality technology, omnidirectional image (ODI) rescaling techniques are increasingly embraced for reducing transmitted and stored file sizes while preserving high image quality. Despite this progress, current ODI rescaling methods predominantly focus on enhancing the quality of images in equirectangular projection (ERP) format, which overlooks the fact that the content viewed on head mounted displays (HMDs) is actually a rendered viewport instead of an ERP image. In this work, we emphasize that focusing solely on ERP quality results in inferior viewport visual experiences for users. Thus, we propose ResVR, which is the first comprehensive framework for the joint Rescaling and Viewport Rendering of ODIs. ResVR allows obtaining LR ERP images for transmission while rendering high-quality viewports for users to watch on HMDs. In our ResVR, a novel discrete pixel sampling strategy is developed to tackle the complex mapping between the viewport and ERP, enabling end-to-end training of ResVR pipeline. Furthermore, a spherical pixel shape representation technique is innovatively derived from spherical differentiation to significantly improve the visual quality of rendered viewports. Extensive experiments demonstrate that our ResVR outperforms existing methods in viewport rendering tasks across different fields of view, resolutions, and view directions while keeping a low transmission overhead.

Primary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: Inspired by the rapid advancements in virtual reality (VR) and augmented reality (AR) technologies, our work introduces ResVR, a novel framework tailored for the joint rescaling and viewport rendering of Omnidirectional Images (ODIs). This framework addresses two core challenges in the domain of immersive multimedia applications: image quality and efficient transmission. Traditional approaches in ODI rescaling have been primarily focused on improving the quality of images in equirectangular projection (ERP) format. However, ResVR innovatively shifts the focus towards the actual viewing experience on head-mounted displays (HMDs), which involves not the ERP image but a rendered viewport. By directly optimizing the quality of viewports, ResVR offers a significant enhancement in immersive media scenarios, tackling the inherent issues associated with existing methods. This approach not only leads to a reduction in the size of files transmitted and stored but also significantly improves the visual quality of content consumed in VR environments. Our work emphasizes immersive media, offering a comprehensive solution that surpasses conventional ODI rescaling techniques in multimedia and multimodal processing, particularly within VR and AR scenarios.

Submission Number: 88