VideoPANDA: Panoramic Video Diffusion with Multi-view Attention

ICLR 2025 submission

Abstract

Recent breakthroughs in generative models have sparked unprecedented interest in a wide range of downstream applications, particularly in the realm of augmented and virtual reality, where immersive content creation is paramount. While significant progress has been made in generating high quality images and videos, creating coherent panoramic videos still remains an open challenge. In this work, we introduce VideoPanda, a novel approach for synthesizing 360-degree videos conditioned on text or video data. VideoPanda leverages multi-view attention layers to augment a video diffusion model, enabling it to be multi-view aware and capable of generating immersive panoramic content. Our model is jointly trained using three conditions: text-only, single-view video and single-view video with multi-view images. This unified training allows for video synthesis from text and video as well as long video generation using the same model trained once. Extensive evaluations on the WEB360 dataset, demonstrate that VideoPanda generates more realistic and coherent 360-degree panoramas across all input conditions compared to existing methods.

Text-Condition Generation

Pano Video

Prompt: A view of snow-covered mountains, a partially frozen body of water, and floating ice chunks. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_natgeo1_15000/default/360__Antarctica___Unexpected_Snow___National_Geographic_t005_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of a tree trunk surrounded by fallen leaves, with branches extending outwards and a blurred background suggesting a garden or park setting. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_natgeo1_15000/default/360__Baby_Pandas___Nat_Geo_WILD_t007_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of towering snow-covered rock formations under a mesmerizing display of green auroras in the night sky. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Northern_Lights__Manpupuner_Rock_Formations._8K_360_video_t002_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of snow-covered landscapes interspersed with water bodies, with a prominent church spire in the foreground. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Church_of_the_Intercession_on_the_River_Nerl_in_spring_flood._Aerial_360_video_in_16K_t010_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of a serene landscape with a church at its center, surrounded by water, with trees reflecting on the calm water surface and a city skyline in the distance. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Church_of_the_Intercession_on_the_River_Nerl_in_spring_flood._Aerial_360_video_in_16K_t003_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of the Church of the Intercession on the Nerl, with its iconic green onion-shaped dome, situated near a flooded river, surrounded by snow-covered landscapes and distant settlements. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Church_of_the_Intercession_on_the_River_Nerl_in_spring_flood._Aerial_360_video_in_16K_t008_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of the vast, pristine salt flats reflecting the azure sky above. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t003_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of the vast salt flat meeting the calm waters, under a clear blue sky. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t005_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of a vast, serene salt flat with a solitary cyclist riding across it. The sky above is vast and filled with clouds, reflecting the hues of a setting or rising sun. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t006_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of three horses grazing in a vast field under a cloudy sky. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t012_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of a vast yellow flower field stretching to the horizon, with a line of green trees and mountains in the background under a partly cloudy sky. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Flower_Field_360__Meditation__in_10K_t003_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of the turbulent river, surrounded by lush greenery and tall palm trees, with distant mountains and a cloudy sky overhead. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Hacha_Falls__Canaima_Lagoon__Venezuela._360_video_in_5K._t001_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of the majestic stupa adorned with prayer flags, surrounded by smaller structures and bustling with devotees. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Swayambhunath__the_Monkey_Temple__Kathmandu__Nepal._16K_360_video._t001_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of a sprawling cityscape during sunset, with lush greenery in the foreground and a setting sun casting a warm hue over the horizon. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Swayambhunath__the_Monkey_Temple__Kathmandu__Nepal._16K_360_video._t002_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of the serene beach with gentle waves crashing onto the rocky shore, under a vast sky painted with hues of orange and blue. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Underwater_Wonders__Red_Sea__Egypt._8K_360_video_t002_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of a vast icy expanse under a cloudy sky. Overlayed text reads: 'ONLY A FEW SPECIALIZED SHIPS CAN CUT THEIR WAY THROUGH THE ICE.' (file: html/processed_results/text_cond/apano_0035_360__Antarctica___Journey_Through_The_Ice___National_Geographic_t003_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of Bryce Canyon National Park, showcasing its unique hoodoos and a rapidly changing landscape. (file: html/processed_results/text_cond/apano_0055_360__Bryce_Canyon___National_Geographic_t001_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of the majestic rock formations, dense forests, and vast open land under a clear blue sky. (file: html/processed_results/text_cond/apano_0058_360__Bryce_Canyon___National_Geographic_t003_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of the vast blue ocean under a clear sky. On the right side, there's a boat named 'ASANTE' with two individuals onboard. One person is seated on the boat, and the other is in the cabin. There's a caption that reads, '8 MILES OFF SHORE SITS A LINE OF AGING OIL RIGS'. (file: html/processed_results/text_cond/apano_0082_360__Dive_Through_an_Oil_Rig_Ecosystem___National_Geographic_t002_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of a vast landscape with snow-covered patches amidst water bodies, surrounded by trees, and a distant cityscape on the horizon. (file: html/processed_results/text_cond/apano_0125_Church_of_the_Intercession_on_the_River_Nerl_in_spring_flood._Aerial_360_video_in_16K_t011_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of the vibrant coral reefs bathed in sunlight, teeming with marine life, and the clear blue waters above. (file: html/processed_results/text_cond/apano_0132_Underwater_Wonders__Red_Sea__Egypt._8K_360_video_t044_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of the multi-tiered pagoda surrounded by lush greenery, with a serene body of water in the foreground and a picturesque sunset in the background. (file: html/processed_results/text_cond/apano_0187_Natural_Wonders_in_Bali._Relaxation_VR_video_in_16K_t002_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of snow-covered mountains, a serene lake, a winding road, and a cluster of houses nestled amidst nature. (file: html/processed_results/text_cond/apano_0185_Lofoten_Islands__Norway._Teaser._8K_360_video._t006_sample_pano/pano_fix.mp4)

Pano Video

Prompt: A view of the sun setting over a dense forest, with the sky painted in hues of orange, blue, and purple. (file: html/processed_results/text_cond/apano_0076_360__Climbing_Giants___National_Geographic_t013_sample_pano/pano_fix.mp4)

Video-Condition Generation

Cond Video

Pano Video

Prompt: A view of a panda lying on the grass, another panda standing nearby, a wooden platform with a panda resting on it, and tall trees surrounding the area. (file: html/processed_results/video_cond/apano_0013_360__Baby_Pandas___Nat_Geo_WILD_t006_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a red sports car driving on a winding road surrounded by lush greenery and tall trees. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/col_cond_sora/merged_id004_start000_end005_sim16.51/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a vibrant underwater coral reef, a winding road that seems to float above the water, and a red sports car driving along the road, giving an illusion of speeding through the underwater world. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/col_cond_sora/merged_id017_start017_end022_sim11.79/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a snow-covered road surrounded by tall pine trees, with a red sports car driving on it. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/col_cond_sora/merged_id030_start027_end037_sim11.76/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a quaint town nestled between rolling hills, with wooden houses, a winding river, and a clear blue sky overhead. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/col_cond_sora/merged_id063_start076_end080_sim16.60/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of towering granite cliffs, a cascading waterfall, a dense forest, and a river flowing through the valley. (file: html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/col_cond_sora/merged_id085_start107_end111_sim17.33/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a serene landscape during sunset, with a church or temple situated on an island amidst a flooded area, surrounded by snow-covered fields. (file: html/processed_results/video_cond/apano_0096_Church_of_the_Intercession_on_the_River_Nerl_in_spring_flood._Aerial_360_video_in_16K_t001_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a serene landscape with a church at its center, surrounded by water, with trees reflecting on the calm water surface and a setting sun in the background. (file: html/processed_results/video_cond/apano_0097_Church_of_the_Intercession_on_the_River_Nerl_in_spring_flood._Aerial_360_video_in_16K_t002_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of the vibrant coral reefs under the clear blue waters of the ocean. (file: html/processed_results/video_cond/apano_0101_Underwater_Wonders__Red_Sea__Egypt._8K_360_video_t040_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a long, straight road stretching through a vast, open field with sparse vegetation. (file: html/processed_results/video_cond/apano_0109_Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t001_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of the vast salt flats, the serene sky with scattered clouds, and the mirror-like reflection on the salt flat's surface. (file: html/processed_results/video_cond/apano_0117_Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t002_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of several horses grazing in a vast field under a cloudy sky. (file: html/processed_results/video_cond/apano_0126_Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t010_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a vast grassy field with two horses grazing. The horse on the left is white, while the one on the right is brown. (file: html/processed_results/video_cond/apano_0127_Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t011_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of the vast expanse of the sea meeting the horizon, with intricate patterns of land and water formations in the foreground, and a serene sky painted with hues of pink and blue above. (file: html/processed_results/video_cond/apano_0131_Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t015_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of the turbulent river, surrounded by lush greenery and tall palm trees, with distant mountains and a cloudy sky overhead. (file: html/processed_results/video_cond/apano_0138_Hacha_Falls__Canaima_Lagoon__Venezuela._360_video_in_5K._t001_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of the turbulent river, the cascading waterfalls, the lush green hills, and the vast sky with scattered clouds. (file: html/processed_results/video_cond/apano_0139_Hacha_Falls__Canaima_Lagoon__Venezuela._360_video_in_5K._t002_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of the turbulent river, surrounded by lush greenery, with mountains in the distance under a partly cloudy sky. (file: html/processed_results/video_cond/apano_0140_Hacha_Falls__Canaima_Lagoon__Venezuela._360_video_in_5K._t003_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of the historic Ipatiev Monastery, with its white walls, green domes, and golden crosses, surrounded by a protective wall and overlooking a serene river. (file: html/processed_results/video_cond/apano_0142_Kostroma._Golden_Ring_of_Russia._360_video_teaser_in_11K_t002_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a serene lake surrounded by autumn trees, with a town in the distance and a church spire on the left. (file: html/processed_results/video_cond/apano_0145_Kostroma._Golden_Ring_of_Russia._360_video_teaser_in_11K_t005_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a snow-covered cityscape with modern buildings, a harbor filled with boats, and majestic snow-capped mountains in the background during sunset. (file: html/processed_results/video_cond/apano_0152_Lofoten_Islands__Norway._Teaser._8K_360_video._t004_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of towering snow-covered rock formations under a mesmerizing display of the Northern Lights. (file: html/processed_results/video_cond/apano_0159_Northern_Lights__Manpupuner_Rock_Formations._8K_360_video_t001_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of the majestic stupa adorned with prayer flags, surrounded by smaller structures and bustling with devotees. (file: html/processed_results/video_cond/apano_0161_Swayambhunath__the_Monkey_Temple__Kathmandu__Nepal._16K_360_video._t001_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of the serene pathway adorned with colorful statues of Buddha, flanked by lush greenery and prayer flags, as visitors walk up the stairs. (file: html/processed_results/video_cond/apano_0171_Swayambhunath__the_Monkey_Temple__Kathmandu__Nepal._16K_360_video._t010_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of the rocky shore, two inflatable boats, and the vast expanse of the sea under a cloudy sky. (file: html/processed_results/video_cond/apano_0005_360__Antarctica___Unexpected_Snow___National_Geographic_t006_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of the rocky shore, two inflatable boats, and a person crouching near the water. (file: html/processed_results/video_cond/apano_0006_360__Antarctica___Unexpected_Snow___National_Geographic_t007_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a panda's face peeking from behind a tree branch amidst lush green foliage. (file: html/processed_results/video_cond/apano_0016_360__Baby_Pandas___Nat_Geo_WILD_t009_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a panda resting on a rock amidst a forest setting. (file: html/processed_results/video_cond/apano_0026_360__Baby_Pandas___Nat_Geo_WILD_t018_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a vast icy expanse with broken ice floes, a ship's bow on the right, and a horizon with a hint of land and a cloudy sky. (file: html/processed_results/video_cond/apano_0023_360__Antarctica___Journey_Through_The_Ice___National_Geographic_t004_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of an offshore oil rig, set against a serene backdrop of calm waters and a radiant sunrise or sunset. (file: html/processed_results/video_cond/apano_0061_360__Dive_Through_an_Oil_Rig_Ecosystem___National_Geographic_t004_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of icy landscapes, floating icebergs, a group of penguins on an ice sheet, snow-covered hills, and a vast expanse of water under a partly cloudy sky. (file: html/processed_results/video_cond/apano_0067_360__Antarctica___Journey_Through_The_Ice___National_Geographic_t008_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of icy mountains, a body of water with floating ice, and a foggy atmosphere. (file: html/processed_results/video_cond/apano_0078_360__Antarctica___Unexpected_Snow___National_Geographic_t001_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of the ASANTE boat navigating near an offshore oil rig. (file: html/processed_results/video_cond/apano_0092_360__Dive_Through_an_Oil_Rig_Ecosystem___National_Geographic_t032_sample_pano/pano_fix.mp4)

Autoregressive Generation

Cond Video

Pano Video

Prompt: A FPV drone shot through a castle on a cliff (file: html/processed_results/autoregressive_flatten/apano_0029_A FPV drone shot through a castle on a cliff/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A dream-like FPV hyper-speed fly through multiple locations. streets at night, a dense tropical jungle, underwater coral (file: html/processed_results/autoregressive_flatten/apano_0052_A dream-like FPV hyper-speed fly through multiple locations. streets at night, a dense tropical jungle, underwater coral/pano_fix.mp4)

Cond Video

Pano Video

Prompt: Handheld tracking shot at night, following a dirty blue ballon floating above the ground in abandon old European street (file: html/processed_results/autoregressive_flatten/apano_0045_Handheld tracking shot at night, following a dirty blue ballon floating above the ground in abandon old European street/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A Japanese animated film of a young woman standing on a ship and looking back at camera (file: html/processed_results/autoregressive_flatten/apano_0038_A Japanese animated film of a young woman standing on a ship and looking back at camera/pano_fix.mp4)

Cond Video

Pano Video

Prompt: gunung bromo (file: html/processed_results/autoregressive_flatten/apano_0005_gunung bromo/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A yellow bird sitting on a branch in a lush jungle with a thick rainforest in the background with mist. (file: html/processed_results/autoregressive_flatten/apano_0077_A yellow bird sitting on a branch in a lush jungle with a thick rainforest in the background with mist./pano_fix.mp4)

Cond Video

Pano Video

Prompt: An empty warehouse dynamically transformed by flora that explode from the ground (file: html/processed_results/autoregressive_flatten/apano_0059_An empty warehouse dynamically transformed by flora that explode from the ground/pano_fix.mp4)

Cond Video

Pano Video

Prompt: A view of a man standing on the bow of a white boat, surrounded by clear blue waters and distant landforms. (file: html/processed_results/autoregressive_flatten/apano_0013_360__Underwater_National_Park___National_Geographic_t001_sample_pano/pano_fix.mp4)

Cond Video

Pano Video

Prompt: Make the scene dramatic with thunders all over, and fires blowing, depicting a terrific sound thundering across the sky and the earth on the battlefield. The scene is dark, intense, and colorful (file: html/processed_results/autoregressive_flatten/apano_0003_Make the scene dramatic with thunders all over, and fires blowing, depicting a terrific sound thundering across the sky and the earth on the battlefield. The scene is dark, intense, and colorful/pano_fix.mp4)

Ours vs. 360DVD

Ours

360DVD

Prompt: A view of the Church of the Intercession on the Nerl, with its iconic green onion-shaped dome, situated near a flooded river, surrounded by snow-covered landscapes and distant settlements.

Ours

360DVD

Prompt: A view of snow-covered mountains, a partially frozen body of water, and floating ice chunks.

Ours

360DVD

Prompt: A view of a tree trunk surrounded by fallen leaves, with branches extending outwards and a blurred background suggesting a garden or park setting.