AR-Light: Enabling Fast and Lightweight Multi-User Augmented Reality via Semantic Segmentation and Collaborative View Synchronization

Yu Wen, Aamir Bader Shah, Ruizhi Cao, Chen Zhang, Jiefu Chen, Xuqing Wu, Chenhao Xie, Xin Fu

Abstract: Multi-user Augmented Reality (MuAR) allows multiple users to interact with shared virtual objects, facilitated by exchanging environment information. Current MuAR systems rely on 3D point clouds for real-world analysis, view synchronization, object rendering, and movement tracking. However, the complexity of 3D point clouds leads to significant processing delays, with approximately 80% of overhead in commercial frameworks. This hampers usability and degrades user experience. Our analysis reveals that maintaining the facing side of the real-world scene in a stable environment provides sufficient information for virtual object placement and rendering. To address this, we introduce a lightweight quadtree structure, representing 2D scenes through semantic segmentation and geometry, as an alternative to 3D point clouds. Additionally, we propose a novel correction method to handle potential shifts in virtual object placement during view synchronization among users. Combining all designs, we implement a fast and lightweight MuAR framework named AR-Light and test our framework on commercial AR devices. The evaluation results on real-world applications demonstrate that AR-Light can achieve high performance in various real-world scenes while maintaining a comparable virtual object placement accuracy.