1000FPS+ Novel View Synthesis from End-to-End Opaque Triangle Optimization

Zhiwen Yan, Weng Fei Low, Tianxin Huang, Gim Hee Lee

Keywords: Novel View Synthesis

Abstract:

Recent implicit and primitive-based radiance field methods, such as NeRF and 3DGS, have demonstrated impressive capabilities in novel view synthesis from multi-view images. However, their custom representations are often incompatible with conventional graphics pipelines, limiting their application in areas like editing, relighting, physics simulation, and particle effects. Additionally, their volume rendering approach requires alpha-blending multiple colors per pixel, which slows down rendering. Traditional differentiable rendering methods, while offering higher compatibility and speed, rely on known mesh topology, making them unsuitable for complex scene-level reconstruction.
To address these limitations, we introduce a novel end-to-end optimization process of disjoint opaque triangles, natively compatible with standard graphics engines for a wide range of applications. To enable gradient-based optimization over the highly non-differentiable rasterization process, we employ a 2D SDF approximation and a two-layer occlusion approximation. We also incorporate density controls to ensure detailed and complete scene reconstructions. Our paper tackles the challenging end-to-end optimization of scene-level novel view synthesis with opaque representation only.
Our approach achieves over 1000 FPS rendering on a single desktop GPU, providing high compatibility and similar novel view synthesis quality to existing methods.

Submission Number: 274