FreeVS: Generative View Synthesis on Free Driving Trajectory
Keywords: Novel View Synthesis, Driving Scene, Free Trajectory, Image Generation
TL;DR: We propose FreeVS, a novel fully generative approach that can synthesize camera views on free new trajectories in real driving scenes.
Abstract: Existing reconstruction-based novel view synthesis methods for driving scenes focus on synthesizing camera views along the recorded trajectory of the ego vehicle.
Their image rendering performance will severely degrade on viewpoints falling out of the recorded trajectory, where camera rays are untrained.
We propose FreeVS, a novel fully generative approach that can synthesize camera views on free new trajectories in real driving scenes.
To control the generation results to be 3D consistent with the real scenes and accurate in viewpoint pose, we propose the pseudo-image representation of view priors to control the generation process.
Viewpoint translation simulation is applied on pseudo-images to simulate camera movement in each direction.
Once trained, FreeVS can be applied to any validation sequences without reconstruction process and synthesis views on novel trajectories.
Moreover, we propose two new challenging benchmarks tailored to driving scenes, which are novel camera synthesis and novel trajectory synthesis, emphasizing the freedom of viewpoints.
Given that no ground truth images are available on novel trajectories, we also propose to evaluate the consistency of images synthesized on novel trajectories with 3D perception models.
Experiments on the Waymo Open Dataset show that FreeVS has a strong image synthesis performance on both the recorded trajectories and novel trajectories.
The code will be released.
Supplementary Material: zip
Primary Area: applications to computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 289
Loading