WonderFree: Enhancing 3D World Generation via Video Diffusion Prior with Multi-view Consistency

Chaojun Ni, Jie Li, Haoyun Li, Hengyu Liu, Xiaofeng Wang, Zheng Zhu, Guosheng Zhao, Boyuan Wang, Chenxin Li, Guan Huang, Wenjun Mei

Keywords: 3D scene generation; Coss-view consistency; Video restoration

Abstract: 3D scene generation from a single image has gained significant attention due to its potential to create immersive virtual worlds. However, a key challenge in current 3D generation methods is the limited explorability, which cannot render high-quality images during larger maneuvers beyond the original viewpoint, particularly when attempting to move forward into unseen areas. To address this challenge, we propose WonderFree, a model that enables users to generate 3D worlds with enhanced freedom to explore from diverse angles and directions. Specifically, we decouple this challenge into two key subproblems: novel view quality, which addresses visual artifacts and floating issues in novel views, and cross-view consistency, which ensures spatial consistency across different viewpoints. To enhance rendering quality in novel views, we introduce WorldRestorer, a data-driven video restoration model designed to eliminate floaters and artifacts. In addition, a data collection pipeline is presented to automatically gather training data for WorldRestorer, ensuring it can handle scenes with varying styles needed for 3D scene generation. Furthermore, to improve cross-view consistency, we propose ConsistView, a multi-view joint restoration mechanism that simultaneously restores multiple perspectives while maintaining spatiotemporal coherence. Qualitative visualization results demonstrate that WonderFree not only enhances rendering quality across diverse viewpoints but also improves global coherence and consistency. These improvements are further confirmed by CLIP-based metrics and a user study showing a 77.20\% preference for WonderFree over WonderWorld.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 8287