Learning 3D Volume Cloud from Single Image

Yuhang Cheng, Yu Zhang, Xiaogang Wang

Abstract: Three-dimensional (3D) cloud modeling plays a pivotal role in advancing atmospheric models and enhancing natural phenomena visualization systems. Nevertheless, the high-quality reconstruction of clouds remains a significant challenge, primarily due to their inherent heterogeneous nature as volumetric media. Image-based modeling approaches offer a promising solution to this challenge. This paper presents a novel two-stage neural network architecture for 3D cloud reconstruction from single image. The first stage introduces an innovative view synthesis network built upon Stable Diffusion, incorporating two specialized modules: a cloud mapper and a viewpoint mapper, which collaboratively generate novel perspective views from a single input image. The second stage implements a physics-based differentiable rendering framework to construct a 3D cloud reconstruction network, leveraging the synthesized multi-view images to optimize a volumetric density grid representation. To enhance the reconstruction fidelity, we integrate real-world cloud density distribution statistics and implement a post-processing refinement using Perlin-Worley noise combined with Fractal Brownian Motion (FBM) for erosion effects. Additionally, to mitigate the inherent limitations of geometric information extraction from single-view images, we developed a comprehensive cloud simulation dataset for pre-training the viewpoint mapper module. This dataset encompasses multi-view cloud images with corresponding camera extrinsic parameters, capturing a diverse range of cloud formations. Extensive quantitative evaluations and qualitative assessments demonstrate the efficacy and potential of our proposed two-stage network in achieving accurate 3D cloud reconstruction from single-view images.

External IDs:dblp:conf/mir/ChengZ025