LiteReality: Graphic-Ready 3D Scene Reconstruction from RGB-D Scans

Zhening Huang, Xiaoyang Wu, Fangcheng Zhong, Hengshuang Zhao, Matthias Nießner, Joan Lasenby

Keywords: 3D Reconstruction; Appearance Modeling; Graphics-Ready Reconstruction

TL;DR: LiteReality is an automatic pipeline that converts RGB-D scans of indoor environments into graphics-ready 3D scenes, featuring high-quality meshes with realistic PBR materials and articulated objects.

Abstract: We propose LiteReality, a novel pipeline that converts RGB-D scans of indoor environments into compact, realistic, and interactive 3D virtual replicas. LiteReality not only reconstructs scenes that visually resemble reality but also supports key features essential for graphics pipelines, such as object individuality, articulation, and high-quality physically based rendering (PBR) materials. At its core, LiteReality first performs scene understanding and parses the results into a coherent 3D layout and object set with the help of a structured scene graph. It then reconstructs the scene by retrieving the most visually similar artist-crafted 3D models from a curated asset database. The Material Painting module further enhances realism by recovering high-quality, spatially varying materials from observed images. Finally, the reconstructed scene is integrated into a simulation engine, where basic physical properties are assigned to enable interactive behavior. The resulting scenes are compact, editable, and fully compatible with standard graphics pipelines, making them suitable for applications in AR/VR, gaming, robotics, and digital-twin systems. In addition, LiteReality introduces a training-free object-retrieval module that achieves state-of-the-art similarity performance on the Scan2CAD dataset, along with a robust Material Painting module capable of transferring appearances from images of any style to 3D assets—even under severe misalignment, occlusion, and poor lighting. We demonstrate the effectiveness of LiteReality on both real-world scans and standard public datasets.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Submission Number: 9604