SmokeSeer: 3D Gaussian Splatting for Smoke Removal and Scene Reconstruction

Neham Jain, Andrew Jong, Sebastian Scherer, Ioannis Gkioulekas

Keywords: gaussian splatting, 3D Reconstruction, thermal imaging

TL;DR: SmokeSeer is a multi-stage 3D inverse rendering framework that fuses RGB and thermal imagery via 3D Gaussian splatting to decompose a scene into the surface part and the non smoke part of the scene.

Abstract:

The presence of smoke in real-world scenes can severely degrade the quality of images and hamper visibility.
Recently introduced methods for image restoration either rely on data-driven priors that are susceptible to hallucination, or are limited to static low-density smoke.
We introduce SmokeSeer, a method for performing simultaneous 3D scene reconstruction and smoke removal from a video capturing multiple views of a scene. To achieve this task, our method uses thermal and RGB images,
leveraging the fact that the reduced scattering in thermal images enables us to see through the smoke.
We build upon 3D Gaussian splatting to fuse information from the two image modalities,
and decompose the scene explicitly into smoke and non-smoke components.
Unlike prior approaches, SmokeSeer handles a broad range of smoke densities and can adapt to temporally varying smoke.
We validate our approach on synthetic data and introduce a new real-world multi-view smoke dataset with RGB and thermal images. We will make code and data publicly available upon publication.

Submission Number: 249