Seeing Video Through Optical Scattering Media using Spatio-Temporal Diffusion Models

Taesung Kwon, Gookho Song, Yoosun Kim, Jong Chul Ye, Mooseok Jang

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Optics, Inverse scattering problem, Spatiotemporal reconstruction, Dynamic scattering media

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose using temporal correlations between time-sequential frames in solving the inverse scattering problem.

Abstract: Optical scattering causes light rays to deviate from their trajectory, posing challenges for imaging through scattering media such as fog and biological tissues. Although diffusion models have been extensively studied for various inverse problems in recent years, its extension to video recovery, especially through highly scattering media, has been an open problem due to the lack of a closed-form forward model and the difficulty of exploiting the spatio-temporal correlation. To address this, here we present a novel inverse scattering solver using a video diffusion model. In particular, by deriving a closed-form forward model from the shower-curtain effect in a dynamic scattering medium, we develop a video diffusion posterior sampling scheme using a diffusion model with temporal attention that maximally exploits the statistical correlation between a series of frames and a series of scattered signals. Unlike previous end-to-end approaches only relied on spatial correlation between a scene and a scattered signal at a specific time point, the adaptability of the proposed method is highly extendable to various types of scenes, various thicknesses of scattering media, and varying distances between a target scene and a medium. In particular, the use of temporal correlation is shown to be critical to faithfully retrieve high-frequency components which are often missed by inverse operations only in spatial domain. Experimental results using the video datasets of moving sperm cells verify the effectiveness of the proposed method. To the best of our knowledge, this is the first video diffusion model to jointly utilize the correlations in both spatial and temporal domains in solving the inverse scattering problem.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7397