OpenTeleView: An Open 3D Teleconferencing Research Platform

Anonymous

Keywords: teleconferencing, Virtual Reality, Encoding-decoding, realtime rendering, avatar

TL;DR: We integrated current state-of-the-art face reconstruction and rendering algorithms into an end-to-end system to illustrate the utility and capabilities of commodity telepresence on a range of 3D displays.

Abstract: Recent demonstrations of 3D telepresence provide a glimpse into a future where 2D video communication is replaced with photo realistic virtual avatars rendered on 3D displays. However, the exist- ing technology demonstrations typically run on expensive dedicated devices that require the calibration of multiple cameras by experts and the underlying reconstruction, compression, transmission, and rendering methods remain proprietary. We describe our open plat- form for real-time end-to-end 3D teleconferencing using commodity hardware coupled with a modular software structure for inserting advanced computer vision algorithms supporting research and development. We demonstrate the utility of our modular end-to-end approach by integrating state-of-the art modules and improving them based on an analysis of current bottlenecks targeting low-latency processing. We include a baseline implementation supporting real-time 3D teleconferencing that provides a new benchmark for evaluation of current and future algorithms. We demonstrate the practicality of our approach with a baseline, a 3D teleconferencing system running at 25 frames per second with 172 ms latency on consumer GPUs that applies to a single RGB camera input and various 3D display technologies. Our 3D teleconferencing platform is open source, which paves the way for computer vision, computer graphics and HCI re- search to continue innovating together to make 3D teleconferencing the telecommunication standard.

Track: HCI/visualization

Revision: No

Revision Reviewers: No opinion