Towards Foveated Rendering For Immersive Remote TeleroboticsDownload PDF

17 Feb 2022, 08:31 (modified: 01 Jul 2022, 18:53)VAM-HRI 2022Readers: Everyone
Keywords: 3D Reconstruction, Virtual Reality, Gaze tracking, Foveated Rendering, Telerobotics
TL;DR: This work discusses the design of an immersive Mixed reality (MR) framework that utilizes the human visual system to facilitate the rendering of 3D reconstructed scenes to enhance teleoperators' sense of presence and immersion in a remote scene.
Abstract: In remote telerobotics applications, real-time 3D perception through RGB-D cameras (real-time 3D reconstruction, point-clouds), and its rendering inside modern virtual reality (VR) environments, can enhance a user’s sense of presence and immersion in a remote scene. However, this approach requires that the whole pipeline from sensor data acquisition to VR rendering satisfy the speed, throughput, and visual quality requirements. Point-cloud data suffers from network latency and throughput limitations that can negatively impact user experience. In this research, the human visual system was taken as an inspiration to address this problem. Human eyes have their sharpest visual acuity at the center of their fieldof-view, which falls off at the periphery. A remote 3D data visualization framework is proposed that utilizes this acuity fall-off to facilitate the processing, transmission, buffering, and rendering in VR of dense point-clouds / 3D reconstructed scenes. The proposed framework shows significant reductions in latency and throughput needs, higher than 60% in both. A preliminary user study shows that the framework does not significantly affect the perceived visual quality
3 Replies