Real-time 3D Human Reconstruction and Rendering System from a Single RGB Camera

Kun Li

Abstract: Transforming 2D human images into 3D appearance is essential for immersive communication. In this paper, we introduce a low-cost real-time 3D human reconstruction and rendering system with a single RGB camera at 28+ FPS, which guarantees both real-time computing speed and realistic rendering results. We use WebRTC to transmit captured images over the Internet for low-latency remote communication. In addition, we design a lighting-robust rendering approach to enhance the lighting perception and generalization ability of the model. Experimental results show that our system achieves high-quality real-time 3D human reconstruction and rendering for different persons wearing various clothes, even with challenging poses. Our system makes use of only a common USB webcam and a consumer-level GPU with real-time performance and high-fidelity results, which provides a consumer-accessible immersive telepresence solution.