SA-ResGS: Self-Augmented Residual 3D Gaussian Splatting for Next Best View Selection

ICLR 2026 Conference Submission15737 Authors

Keywords: 3D Gaussian Splatting, Active view selection, Next Best View, Residual Learning

TL;DR: SA-ResGS enhances next-best-view selection in 3D Gaussian Splatting by combining physically grounded view selection with residual supervision, improving uncertainty estimation and reconstruction under sparse-view settings.

Abstract: We propose Self-Augmented Residual 3D Gaussian Splatting (SA-ResGS), a novel framework for stabilizing uncertainty quantification and enhancing uncertainty-aware supervision in next-best-view selection for active scene reconstruction. SA-ResGS improves both the reliability of uncertainty estimates and their effectiveness for supervision by generating Self-Augmented point clouds (SA-Points) via triangulation between a training view and a rasterized extrapolated view, enabling efficient scene coverage estimation. While improving scene coverage through physically guided view selection, SA-ResGS also addresses the challenge of under-supervised Gaussians, an issue exacerbated by sparse and wide-baseline views, by introducing the first residual learning strategy tailored for 3D Gaussian Splatting. This targeted supervision mechanism enhances gradient flow in high-uncertainty Gaussians by combining uncertainty-driven filtering with dropout- and hard-negative-mining-inspired sampling. Our contributions are threefold: (1) a physically grounded view selection strategy that promotes efficient and uniform scene coverage; (2) an uncertainty-aware residual supervision scheme that amplifies learning signals for weakly contributing Gaussians, improving training stability and uncertainty estimation across scenes with diverse camera distributions; (3) an implicit unbiasing of uncertainty quantification as a consequence of constrained view selection and residual supervision, which together mitigate the conflicting effects of wide-baseline exploration and sparse-view ambiguity in NBV planning. Experiments on Mip-NeRF 360, Deep Blending, and Tanks and Temples datasets demonstrate that SA-ResGS outperforms state-of-the-art baselines in both reconstruction quality and view selection robustness.

Primary Area: applications to robotics, autonomy, planning

Submission Number: 15737