DroneDreamer: Multi-View Low-Altitude World Model with Adaptive Control

Yiding Ma, Chen Gao, Yu Shang, Wei Wu, Yong Li

Keywords: World Model, Low-Altitude UAV, Multi-View Video Generation, Style Transfer

Abstract: Recent advancements in world models have made significant strides in the fields of autonomous driving and robotic motion, showing great potential as the foundation for general artificial intelligence. Although DiT with 3D VAE has made significant progress in this field, existing models have yet to be effectively applied to low-altitude scenarios, still facing challenges such as difficulty in acquiring conditions, unstable sampling, and data scarcity, which render current methods ineffective. In this paper, we introduce a new research problem: Low-Altitude World Models(LAWM), focusing on the applicability of world models in low-altitude scenarios. To address these issues, we propose DroneDreamer, a novel LAWM that incorporates an adaptive viewpoint control mechanism and an image style domain adaptation technique, enabling multi-view conditional generation with limited conditions. Additionally, we construct a drone flight dataset collected from a simulated modeling environment and a data processing pipeline, along with a progressive training strategy to ensure the accuracy of control capabilities and style adaptation, as well as generalizability. Consequently, our model achieves a 58\% improvement in image generation capabilities on the drone flight dataset compared to specialized models, and a 63\% improvement in multi-view consistency compared to combined baseline models. We believe DroneDreamer can provide a foundational world model for the low-altitude UAV domain, benefiting low-altitude navigation tasks, data generation, flight prediction, and inspiring valuable applications.Our code is available at \href{https://anonymous.4open.science/r/DroneDreamer-B629}{https://anonymous.4open.science/r/DroneDreamer-B629}

Primary Area: generative models

Submission Number: 10717