ConsisDrive: Identity-Preserving Driving World Models for Video Generation by Instance Mask

Abstract

Autonomous driving relies on robust models trained on large-scale, high-quality multi-view driving videos. Although world models provide a cost-effective solution for generating realistic driving data, they often suffer from identity drift, where the same object changes its appearance or category across frames due to the absence of instance-level temporal constraints. We introduce ConsisDrive, an identity-preserving driving world model designed to enforce temporal consistency at the instance level. Our framework incorporates two key components: (1) Instance-Masked Attention, which applies instance identity masks and trajectory masks within attention blocks to ensure that visual tokens interact only with their corresponding instance features across spatial and temporal dimensions, thereby preserving object identity consistency; and (2) Instance-Masked Loss, which adaptively emphasizes foreground regions with probabilistic instance masking, reducing background noise while maintaining overall scene fidelity. By integrating these mechanisms, ConsisDrive achieves state-of-the-art driving video generation quality and demonstrates significant improvements in downstream autonomous driving tasks on the nuScenes dataset.

ConsisDrive Overview

1. Instruction Following Ability

1.1 Text Prompt Editing

ConsisDrive can generate diverse weather scenarios from the same control conditions.
It means it is possible to simulate extreme weather conditions for training perception models.
We append descriptors like "sunny," "rainy," or "night" to text prompts for video editing.
The videos below show the "control condition", followed by "sunny", "rainy", and "night" scenarios.

Control Condition

Sunny, Busy street, parked cars, parking lot, ped sitting at bus stop, motorcycle driving.

"Sunny": clear skies with sunlight shining on the scene, reflecting bright and vivid environmental details.

Rainy, Busy street, parked cars, parking lot, ped sitting at bus stop, motorcycle driving.

"Rainy": wet road surfaces and blurred camera views caused by raindrops, adding realistic weather dynamics.

Night, Busy street, parked cars, parking lot, ped sitting at bus stop, motorcycle driving.

"Night": dimly lit scenes with streetlights and reduced visibility, accurately simulating nighttime driving conditions.

2. Stochastic Diversity of Generation

ConsisDrive can generate diverse videos using varying stochastic noise inputs and the same control conditions.
The videos below show the "control condition" and two sampled videos with different stochastic noise inputs in each line.
Both sampled videos in the second and third line adhere to the constraints defined in the first line.

Control Condition

Control Condition

Oncoming bus, parked car on center divider, pedestrian, roundabout, parked trucks.

Sample 1

Ped walking along the road, bus stops with waiting passenger, dense parked cars at the left.

Sample 1

Oncoming bus, parked car on center divider, pedestrian, roundabout, parked trucks.

Sample 2

Ped walking along the road, bus stops with waiting passenger, dense parked cars at the left.

Sample 2

3. Generalization to Private Dataset

In addition to public datasets, we trained on a 200 hour private dataset. The results below demonstrate that we achieved similar generation quality and control performance as on nuScenes, highlighting the generalization capability of our method.

4. Instance Identity Preservation

4.1 Comparison with Baseline

Realistic drive view.

Marching Cube mesh of seg153495

Realistic drive view.

a. Category Shift. In DriveDreamer, the bus (in the third view) gradually turns into a truck, indicating a failure to preserve semantic identity over time. In our method, we preserve instance semantic attributes over time.

Realistic drive view.

Marching Cube mesh of seg153495

Realistic drive view.

b. Color Shift. In MagicDrive-V2, the car's color changes inconsistently across frames, violating temporal appearance consistency. While in our method, it preserves color consistency.

4.2 Instance Attributes Binding && Propagation

a. Category Preservation.

b. Color Preservation.

4.3 Foreground Small Objects Emphasis

ConsisDrive enhances the fidelity of small and challenging objects(e.g.,pedestrians and bicycles).
We overlay the 3D bounding box projections onto the generated videos.

Wait at intersection, peds on sidewalk, turn right, cones, cross intersection.

Small and densely packed objects are precisely rendered at their correct locations, following 3D bounding box coordinates.
Objects track their previous attributes as guided by the Instance Masked Attention, ensuring temporal consistency across frames.