Towards Foveated Rendering For Immersive Remote Telerobotics

Yonas Teodros Tefera, Dario Mazzanti, Sara Anastasi, Darwin Caldwell, Paolo Fiorini, Nikhil Deshpande

Keywords: 3D Reconstruction, Virtual Reality, Gaze tracking, Foveated Rendering, Telerobotics

TL;DR: This work discusses the design of an immersive Mixed reality (MR) framework that utilizes the human visual system to facilitate the rendering of 3D reconstructed scenes to enhance teleoperators' sense of presence and immersion in a remote scene.

Abstract: In remote telerobotics applications, real-time 3D perception through RGB-D cameras (real-time 3D reconstruction, point-clouds), and its rendering inside modern virtual reality (VR) environments, can enhance a user’s sense of presence and immersion in a remote scene. However, this approach requires that the whole pipeline from sensor data acquisition to VR rendering satisfy the speed, throughput, and visual quality requirements. Point-cloud data suffers from network latency and throughput limitations that can negatively impact user experience. In this research, the human visual system was taken as an inspiration to address this problem. Human eyes have their sharpest visual acuity at the center of their fieldof-view, which falls off at the periphery. A remote 3D data visualization framework is proposed that utilizes this acuity fall-off to facilitate the processing, transmission, buffering, and rendering in VR of dense point-clouds / 3D reconstructed scenes. The proposed framework shows significant reductions in latency and throughput needs, higher than 60% in both. A preliminary user study shows that the framework does not significantly affect the perceived visual quality