Incremental3D: Real-time Incremental 3D Scene Generation with Scene Graphs

Penggang GAO, Yonas Teodros Tefera, Darwin G. Caldwell, Nikhil Deshpande

Abstract:

Realistic 3D environments are important for a wide range of applications, including robotics, simulation, virtual reality, and video games.
The goal of 3D scene generation is to create spatially structured, semantically meaningful, and visually realistic environments that capture objects and their relationships in space. Graph-based 3D scene generation approaches represent environments as scene graphs, where nodes correspond to objects and edges encode their semantic and spatial relationships. However, existing methods become inefficient when the 3D scene graph evolves incrementally, because they are fundamentally single-shot: inserting even a single new object requires regenerating the entire scene. This global re-computation incurs prohibitive latency and scalability limitations. To address this limitation, we propose Incremental3D, a framework for incremental 3D scene generation in real-time from evolving scene graphs. Incremental3D augments the scene graph with a global context node that captures a holistic representation of the evolving environment. At each update step, this node aggregates information from new nodes and edges to form a global embedding. Newly inserted objects are then generated by conditioning on both this embedding and their local features, enabling geometry synthesis and spatial prediction without recomputing unchanged regions. Extensive experiments demonstrate that Incremental3D achieves a generation rate of 38 Hz, while maintaining high spatial and geometric accuracy, indicating its potential for real-time and latency-sensitive applications.

Submission Type: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission: Camera-Ready Version

Assigned Action Editor: ~Matthew_Walter1

Submission Number: 6689