VideoPANDA: Panoramic Video Diffusion with Multi-view Attention
ICLR 2025 submission
Abstract
Recent breakthroughs in generative models have sparked unprecedented interest in a wide range of
downstream applications, particularly in the realm of augmented and virtual reality, where immersive
content creation is paramount. While significant progress has been made in generating high quality
images and videos, creating coherent panoramic videos still remains an open challenge. In this work,
we
introduce VideoPanda, a novel approach for synthesizing 360-degree videos conditioned on text or
video
data. VideoPanda leverages multi-view attention layers to augment a video diffusion model, enabling
it
to be multi-view aware and capable of generating immersive panoramic content. Our model is jointly
trained using three conditions: text-only, single-view video and single-view video with multi-view
images. This unified training allows for video synthesis from text and video as well as long video
generation using the same model trained once. Extensive evaluations on the WEB360 dataset,
demonstrate
that VideoPanda generates more realistic and coherent 360-degree panoramas across all input
conditions
compared to existing methods.
Text-Condition Generation
Pano Video
Prompt: A view of snow-covered mountains, a partially frozen body of water, and floating ice
chunks.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_natgeo1_15000/default/360__Antarctica___Unexpected_Snow___National_Geographic_t005_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of a tree trunk surrounded by fallen leaves, with branches extending outwards and
a blurred background suggesting a garden or park setting.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_natgeo1_15000/default/360__Baby_Pandas___Nat_Geo_WILD_t007_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of towering snow-covered rock formations under a mesmerizing display of green
auroras in the night sky.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Northern_Lights__Manpupuner_Rock_Formations._8K_360_video_t002_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of snow-covered landscapes interspersed with water bodies, with a prominent
church spire in the foreground.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Church_of_the_Intercession_on_the_River_Nerl_in_spring_flood._Aerial_360_video_in_16K_t010_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of a serene landscape with a church at its center, surrounded by water, with
trees reflecting on the calm water surface and a city skyline in the distance.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Church_of_the_Intercession_on_the_River_Nerl_in_spring_flood._Aerial_360_video_in_16K_t003_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of the Church of the Intercession on the Nerl, with its iconic green onion-shaped
dome, situated near a flooded river, surrounded by snow-covered landscapes and distant
settlements.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Church_of_the_Intercession_on_the_River_Nerl_in_spring_flood._Aerial_360_video_in_16K_t008_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of the vast, pristine salt flats reflecting the azure sky above.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t003_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of the vast salt flat meeting the calm waters, under a clear blue sky.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t005_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of a vast, serene salt flat with a solitary cyclist riding across it. The sky
above is vast and filled with clouds, reflecting the hues of a setting or rising sun.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t006_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of three horses grazing in a vast field under a cloudy sky.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t012_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of a vast yellow flower field stretching to the horizon, with a line of green
trees and mountains in the background under a partly cloudy sky.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Flower_Field_360__Meditation__in_10K_t003_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of the turbulent river, surrounded by lush greenery and tall palm trees, with
distant mountains and a cloudy sky overhead.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Hacha_Falls__Canaima_Lagoon__Venezuela._360_video_in_5K._t001_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of the majestic stupa adorned with prayer flags, surrounded by smaller structures
and bustling with devotees.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Swayambhunath__the_Monkey_Temple__Kathmandu__Nepal._16K_360_video._t001_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of a sprawling cityscape during sunset, with lush greenery in the foreground and
a setting sun casting a warm hue over the horizon.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Swayambhunath__the_Monkey_Temple__Kathmandu__Nepal._16K_360_video._t002_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of the serene beach with gentle waves crashing onto the rocky shore, under a vast
sky painted with hues of orange and blue.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/text_cond_ytb_pano2_15000/Underwater_Wonders__Red_Sea__Egypt._8K_360_video_t002_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of a vast icy expanse under a cloudy sky. Overlayed text reads: 'ONLY A FEW
SPECIALIZED SHIPS CAN CUT THEIR WAY THROUGH THE ICE.'
(file:
html/processed_results/text_cond/apano_0035_360__Antarctica___Journey_Through_The_Ice___National_Geographic_t003_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of Bryce Canyon National Park, showcasing its unique hoodoos and a rapidly
changing landscape.
(file:
html/processed_results/text_cond/apano_0055_360__Bryce_Canyon___National_Geographic_t001_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of the majestic rock formations, dense forests, and vast open land under a clear
blue sky.
(file:
html/processed_results/text_cond/apano_0058_360__Bryce_Canyon___National_Geographic_t003_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of the vast blue ocean under a clear sky. On the right side, there's a boat named
'ASANTE' with two individuals onboard. One person is seated on the boat, and the other is in the
cabin. There's a caption that reads, '8 MILES OFF SHORE SITS A LINE OF AGING OIL RIGS'.
(file:
html/processed_results/text_cond/apano_0082_360__Dive_Through_an_Oil_Rig_Ecosystem___National_Geographic_t002_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of a vast landscape with snow-covered patches amidst water bodies, surrounded by
trees, and a distant cityscape on the horizon.
(file:
html/processed_results/text_cond/apano_0125_Church_of_the_Intercession_on_the_River_Nerl_in_spring_flood._Aerial_360_video_in_16K_t011_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of the vibrant coral reefs bathed in sunlight, teeming with marine life, and the
clear blue waters above.
(file:
html/processed_results/text_cond/apano_0132_Underwater_Wonders__Red_Sea__Egypt._8K_360_video_t044_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of the multi-tiered pagoda surrounded by lush greenery, with a serene body of
water in the foreground and a picturesque sunset in the background.
(file:
html/processed_results/text_cond/apano_0187_Natural_Wonders_in_Bali._Relaxation_VR_video_in_16K_t002_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of snow-covered mountains, a serene lake, a winding road, and a cluster of houses
nestled amidst nature.
(file:
html/processed_results/text_cond/apano_0185_Lofoten_Islands__Norway._Teaser._8K_360_video._t006_sample_pano/pano_fix.mp4)
Pano Video
Prompt: A view of the sun setting over a dense forest, with the sky painted in hues of orange,
blue, and purple.
(file:
html/processed_results/text_cond/apano_0076_360__Climbing_Giants___National_Geographic_t013_sample_pano/pano_fix.mp4)
Video-Condition Generation
Cond Video
Pano Video
Prompt: A view of a panda lying on the grass, another panda standing nearby, a wooden platform
with a panda resting on it, and tall trees surrounding the area.
(file:
html/processed_results/video_cond/apano_0013_360__Baby_Pandas___Nat_Geo_WILD_t006_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a red sports car driving on a winding road surrounded by lush greenery and
tall trees.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/col_cond_sora/merged_id004_start000_end005_sim16.51/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a vibrant underwater coral reef, a winding road that seems to float above the
water, and a red sports car driving along the road, giving an illusion of speeding through the
underwater world.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/col_cond_sora/merged_id017_start017_end022_sim11.79/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a snow-covered road surrounded by tall pine trees, with a red sports car
driving on it.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/col_cond_sora/merged_id030_start027_end037_sim11.76/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a quaint town nestled between rolling hills, with wooden houses, a winding
river, and a clear blue sky overhead.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/col_cond_sora/merged_id063_start076_end080_sim16.60/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of towering granite cliffs, a cascading waterfall, a dense forest, and a river
flowing through the valley.
(file:
html/processed_results/mvvid_web360_randcond_randomizeMatrix_freeze_dbg1/col_cond_sora/merged_id085_start107_end111_sim17.33/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a serene landscape during sunset, with a church or temple situated on an
island amidst a flooded area, surrounded by snow-covered fields.
(file:
html/processed_results/video_cond/apano_0096_Church_of_the_Intercession_on_the_River_Nerl_in_spring_flood._Aerial_360_video_in_16K_t001_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a serene landscape with a church at its center, surrounded by water, with
trees reflecting on the calm water surface and a setting sun in the background.
(file:
html/processed_results/video_cond/apano_0097_Church_of_the_Intercession_on_the_River_Nerl_in_spring_flood._Aerial_360_video_in_16K_t002_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of the vibrant coral reefs under the clear blue waters of the ocean.
(file:
html/processed_results/video_cond/apano_0101_Underwater_Wonders__Red_Sea__Egypt._8K_360_video_t040_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a long, straight road stretching through a vast, open field with sparse
vegetation.
(file:
html/processed_results/video_cond/apano_0109_Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t001_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of the vast salt flats, the serene sky with scattered clouds, and the mirror-like
reflection on the salt flat's surface.
(file:
html/processed_results/video_cond/apano_0117_Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t002_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of several horses grazing in a vast field under a cloudy sky.
(file:
html/processed_results/video_cond/apano_0126_Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t010_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a vast grassy field with two horses grazing. The horse on the left is white,
while the one on the right is brown.
(file:
html/processed_results/video_cond/apano_0127_Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t011_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of the vast expanse of the sea meeting the horizon, with intricate patterns of
land and water formations in the foreground, and a serene sky painted with hues of pink and blue
above.
(file:
html/processed_results/video_cond/apano_0131_Elton_Lake__the_biggest_mineral_lake_in_Europe._8K_aerial_360_video._t015_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of the turbulent river, surrounded by lush greenery and tall palm trees, with
distant mountains and a cloudy sky overhead.
(file:
html/processed_results/video_cond/apano_0138_Hacha_Falls__Canaima_Lagoon__Venezuela._360_video_in_5K._t001_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of the turbulent river, the cascading waterfalls, the lush green hills, and the
vast sky with scattered clouds.
(file:
html/processed_results/video_cond/apano_0139_Hacha_Falls__Canaima_Lagoon__Venezuela._360_video_in_5K._t002_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of the turbulent river, surrounded by lush greenery, with mountains in the
distance under a partly cloudy sky.
(file:
html/processed_results/video_cond/apano_0140_Hacha_Falls__Canaima_Lagoon__Venezuela._360_video_in_5K._t003_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of the historic Ipatiev Monastery, with its white walls, green domes, and golden
crosses, surrounded by a protective wall and overlooking a serene river.
(file:
html/processed_results/video_cond/apano_0142_Kostroma._Golden_Ring_of_Russia._360_video_teaser_in_11K_t002_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a serene lake surrounded by autumn trees, with a town in the distance and a
church spire on the left.
(file:
html/processed_results/video_cond/apano_0145_Kostroma._Golden_Ring_of_Russia._360_video_teaser_in_11K_t005_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a snow-covered cityscape with modern buildings, a harbor filled with boats,
and majestic snow-capped mountains in the background during sunset.
(file:
html/processed_results/video_cond/apano_0152_Lofoten_Islands__Norway._Teaser._8K_360_video._t004_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of towering snow-covered rock formations under a mesmerizing display of the
Northern Lights.
(file:
html/processed_results/video_cond/apano_0159_Northern_Lights__Manpupuner_Rock_Formations._8K_360_video_t001_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of the majestic stupa adorned with prayer flags, surrounded by smaller structures
and bustling with devotees.
(file:
html/processed_results/video_cond/apano_0161_Swayambhunath__the_Monkey_Temple__Kathmandu__Nepal._16K_360_video._t001_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of the serene pathway adorned with colorful statues of Buddha, flanked by lush
greenery and prayer flags, as visitors walk up the stairs.
(file:
html/processed_results/video_cond/apano_0171_Swayambhunath__the_Monkey_Temple__Kathmandu__Nepal._16K_360_video._t010_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of the rocky shore, two inflatable boats, and the vast expanse of the sea under a
cloudy sky.
(file:
html/processed_results/video_cond/apano_0005_360__Antarctica___Unexpected_Snow___National_Geographic_t006_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of the rocky shore, two inflatable boats, and a person crouching near the water.
(file:
html/processed_results/video_cond/apano_0006_360__Antarctica___Unexpected_Snow___National_Geographic_t007_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a panda's face peeking from behind a tree branch amidst lush green foliage.
(file:
html/processed_results/video_cond/apano_0016_360__Baby_Pandas___Nat_Geo_WILD_t009_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a panda resting on a rock amidst a forest setting.
(file:
html/processed_results/video_cond/apano_0026_360__Baby_Pandas___Nat_Geo_WILD_t018_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a vast icy expanse with broken ice floes, a ship's bow on the right, and a
horizon with a hint of land and a cloudy sky.
(file:
html/processed_results/video_cond/apano_0023_360__Antarctica___Journey_Through_The_Ice___National_Geographic_t004_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of an offshore oil rig, set against a serene backdrop of calm waters and a
radiant sunrise or sunset.
(file:
html/processed_results/video_cond/apano_0061_360__Dive_Through_an_Oil_Rig_Ecosystem___National_Geographic_t004_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of icy landscapes, floating icebergs, a group of penguins on an ice sheet,
snow-covered hills, and a vast expanse of water under a partly cloudy sky.
(file:
html/processed_results/video_cond/apano_0067_360__Antarctica___Journey_Through_The_Ice___National_Geographic_t008_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of icy mountains, a body of water with floating ice, and a foggy atmosphere.
(file:
html/processed_results/video_cond/apano_0078_360__Antarctica___Unexpected_Snow___National_Geographic_t001_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of the ASANTE boat navigating near an offshore oil rig.
(file:
html/processed_results/video_cond/apano_0092_360__Dive_Through_an_Oil_Rig_Ecosystem___National_Geographic_t032_sample_pano/pano_fix.mp4)
Autoregressive Generation
Cond Video
Pano Video
Prompt: A FPV drone shot through a castle on a cliff
(file: html/processed_results/autoregressive_flatten/apano_0029_A FPV drone shot through a
castle on a cliff/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A dream-like FPV hyper-speed fly through multiple locations. streets at night, a dense
tropical jungle, underwater coral
(file: html/processed_results/autoregressive_flatten/apano_0052_A dream-like FPV hyper-speed fly
through multiple locations. streets at night, a dense tropical jungle, underwater
coral/pano_fix.mp4)
Cond Video
Pano Video
Prompt: Handheld tracking shot at night, following a dirty blue ballon floating above the ground
in abandon old European street
(file: html/processed_results/autoregressive_flatten/apano_0045_Handheld tracking shot at night,
following a dirty blue ballon floating above the ground in abandon old European
street/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A Japanese animated film of a young woman standing on a ship and looking back at camera
(file: html/processed_results/autoregressive_flatten/apano_0038_A Japanese animated film of a
young woman standing on a ship and looking back at camera/pano_fix.mp4)
Cond Video
Pano Video
Prompt: gunung bromo
(file: html/processed_results/autoregressive_flatten/apano_0005_gunung bromo/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A yellow bird sitting on a branch in a lush jungle with a thick rainforest in the
background with mist.
(file: html/processed_results/autoregressive_flatten/apano_0077_A yellow bird sitting on a
branch in a lush jungle with a thick rainforest in the background with mist./pano_fix.mp4)
Cond Video
Pano Video
Prompt: An empty warehouse dynamically transformed by flora that explode from the ground
(file: html/processed_results/autoregressive_flatten/apano_0059_An empty warehouse dynamically
transformed by flora that explode from the ground/pano_fix.mp4)
Cond Video
Pano Video
Prompt: A view of a man standing on the bow of a white boat, surrounded by clear blue waters and
distant landforms.
(file:
html/processed_results/autoregressive_flatten/apano_0013_360__Underwater_National_Park___National_Geographic_t001_sample_pano/pano_fix.mp4)
Cond Video
Pano Video
Prompt: Make the scene dramatic with thunders all over, and fires blowing, depicting a terrific
sound thundering across the sky and the earth on the battlefield. The scene is dark, intense,
and colorful
(file: html/processed_results/autoregressive_flatten/apano_0003_Make the scene dramatic with
thunders all over, and fires blowing, depicting a terrific sound thundering across the sky and
the earth on the battlefield. The scene is dark, intense, and colorful/pano_fix.mp4)
Ours vs. 360DVD
Ours
360DVD
Prompt: A view of the Church of the Intercession on the Nerl, with its iconic green onion-shaped
dome, situated near a flooded river, surrounded by snow-covered landscapes and distant
settlements.
Ours
360DVD
Prompt: A view of snow-covered mountains, a partially frozen body of water, and floating ice
chunks.
Ours
360DVD
Prompt: A view of a tree trunk surrounded by fallen leaves, with branches extending outwards and
a blurred background suggesting a garden or park setting.