# DriveGuard Dataset Creation Pipeline

**Complete 4-Step Workflow for Creating Curated Dashcam Video Datasets with Ground Truth**

This pipeline creates high-quality datasets for DriveGuard evaluation through a comprehensive 4-step process: downloading raw videos from YouTube, manually reviewing and marking clips, extracting the final curated dataset with perfect timing alignment, and creating comprehensive ground truth annotations through a web-based annotation tool.

## 🎯 Overview

The dataset creation pipeline implements a complete workflow from raw video acquisition to curated dataset:

### 📥 **Step 1: Video Download**
- **175+ targeted search queries** across 30+ safety-critical categories
- **Intelligent duplicate detection** by video ID and URL across categories  
- **Real-time progress bars** with download status and completion tracking
- **Resume capability** for interrupted downloads without re-downloading existing videos

### 🎬 **Step 2: Manual Review** 
- **Interactive timeline** with drag-and-drop clip marking
- **Real-time video preview** while positioning clips
- **Event type classification** with severity levels
- **Web-based interface** for efficient review workflow

### ✂️ **Step 3: Clip Extraction**
- **Perfect timing alignment** between preview and extracted clips
- **Meaningful filenames** with sequence numbers and categories  
- **Smart resume capability** to avoid re-processing existing clips
- **Comprehensive metadata preservation** from all steps

### 🎯 **Step 4: Ground Truth Annotation**
- **Web-based annotation tool** for creating comprehensive ground truth data
- **5-step pipeline**: Video annotation → Scene extraction → Violation analysis → Accident analysis → Assessment generation
- **Professional video player** with timeline navigation and frame-by-frame controls
- **Manual editing capabilities** for all generated data with real-time validation

### 🤖 **Step 5: System Output Generation**
- **Multi-component pipeline** for automated analysis and evaluation
- **Multi-model processing** with 23 supported models including GPT, Claude, Gemini, and specialized models
- **Resume capability** to skip existing outputs and continue interrupted processing
- **Runtime tracking** for performance monitoring and optimization

### 🚨 **Safety-Critical Focus**

The downloader is specifically designed to capture diverse driving scenarios:
- **Critical Violations** - Red light running, wrong-way driving, school bus violations
- **Aggressive Behaviors** - Road rage, tailgating, dangerous overtaking, cutting off
- **Vulnerable User Conflicts** - Pedestrian, cyclist, and motorcycle safety scenarios  
- **Environmental Hazards** - Weather-related risks, construction zones, night driving
- **Infrastructure Challenges** - Railroad crossings, bridges, tunnels, road debris

## 📁 System Components

| File | Purpose | Key Features |
|------|---------|--------------|
| `s1_youtube_downloader.py` | Advanced YouTube video downloader | • 30+ search categories<br>• Progress tracking<br>• Metadata management<br>• Resume capability<br>• Duplicate detection |
| `s2_video_reviewer/` | Manual video review and clip marking tool | • Interactive timeline<br>• Drag-and-drop clips<br>• Event type classification<br>• Web-based interface |
| `s3_extract_clips.py` | Extract marked clips to final dataset | • Perfect timing alignment<br>• Meaningful filenames<br>• Metadata preservation<br>• Resume capability |
| `s4_annotate_ground_truth/` | Web-based ground truth annotation tool | • 5-step annotation pipeline<br>• Professional video player<br>• Manual editing interface<br>• FastAPI backend with REST API |
| `s5_generate_system_output/` | Automated system output generation | • Multi-component processing<br>• 23 supported AI models<br>• Runtime tracking<br>• Resume capability |
| `README.md` | Documentation and usage instructions | • Setup guide<br>• Category reference<br>• Usage examples<br>• Command reference |

## ✨ Key Features

### 📊 **Progress Tracking**
- **Real-time Progress Bars**: Visual progress for overall download and individual categories
- **Status Updates**: Live display of current video being downloaded with completion status
- **Download Statistics**: Total videos, file sizes, completion rates, and time estimates

### 🔄 **Resume & Checkpoint System**
- **Intelligent Resume**: Continue interrupted downloads without re-downloading existing videos
- **Checkpoint Management**: Automatic saving of progress after each successful download
- **Parameter Validation**: Ensures resume parameters match original download configuration
- **File Verification**: Checks that video files actually exist before skipping downloads

### 📋 **Comprehensive Metadata**
- **Video Information**: Title, URL, ID, duration, view count, upload date, relevance score
- **File Details**: Path, size, download timestamp, category organization
- **Export Capability**: JSON export of all video metadata for external analysis
- **Search Integration**: Metadata helps avoid downloading duplicate content across searches

### 🚫 **Duplicate Prevention**
- **Multi-level Detection**: Prevents duplicates by video ID, URL, and cross-category checking
- **Smart Skipping**: Automatically skips previously downloaded videos with status feedback
- **File Integrity**: Verifies existing files before marking as already downloaded

## 🚀 Quick Start

### Prerequisites

```bash
# Install system dependencies
brew install ffmpeg  # macOS
# OR: apt install ffmpeg  # Linux

# Install Python dependencies
uv sync  # yt-dlp and tqdm are included as dependencies
```

### Basic Usage

```bash
# From project root directory

# Step 1: Download 50 videos total (distributed across categories)
uv run evaluation/make_dataset/s1_youtube_downloader.py --total-videos 50

# Download 3 videos per category (traditional method)
uv run evaluation/make_dataset/s1_youtube_downloader.py --videos-per-category 3

# Resume interrupted download
uv run evaluation/make_dataset/s1_youtube_downloader.py --resume --total-videos 50

# Check download status
uv run evaluation/make_dataset/s1_youtube_downloader.py --status

# List all available categories
uv run evaluation/make_dataset/s1_youtube_downloader.py --list-categories

# Export video metadata to JSON
uv run evaluation/make_dataset/s1_youtube_downloader.py --export-metadata

# Step 2: Review and mark clips
cd evaluation/make_dataset/s2_video_reviewer/
uv run main.py
cd ..

# Step 3: Extract marked clips to final dataset
uv run evaluation/make_dataset/s3_extract_clips.py --status
uv run evaluation/make_dataset/s3_extract_clips.py

# Step 4: Create ground truth annotations (web interface)
cd evaluation/make_dataset/s4_annotate_ground_truth/
uv run python -m evaluation.make_dataset.s4_annotate_ground_truth.app
# Access at http://localhost:8000

# Step 5: Generate system outputs for evaluation
cd evaluation/make_dataset/s5_generate_system_output/

# Generate scene extraction outputs for all models
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --component scene

# Generate violation checking outputs for all models  
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --component violation

# Generate for a specific model only
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --component violation --model "openai:gpt-4o"

# Check status of system outputs
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --status

# Step 3 advanced options
uv run evaluation/make_dataset/s3_extract_clips.py --force  # Re-extract existing clips
uv run evaluation/make_dataset/s3_extract_clips.py --categories accidents_general,road_rage  # Specific categories
```

### Command Line Options

| Parameter | Description | Example |
|-----------|-------------|---------|
| `--total-videos N` | Download N videos total, distributed across categories | `--total-videos 100` |
| `--videos-per-category N` | Download N videos per category (default: 3) | `--videos-per-category 5` |
| `--resume` | Resume interrupted download without re-downloading | `--resume --total-videos 50` |
| `--status` | Show current download status and statistics | `--status` |
| `--list-categories` | List all available video categories | `--list-categories` |
| `--export-metadata [FILE]` | Export video metadata to JSON file | `--export-metadata my_videos.json` |
| `--output-dir DIR` | Custom output directory | `--output-dir ./my_videos` |

## 🔧 Step 3: Clip Extraction (Detailed)

### Prerequisites for Step 3

```bash
# Ensure ffmpeg is installed
brew install ffmpeg  # macOS
# OR: apt install ffmpeg  # Linux

# Verify previous steps completed
ls data/data_prepare/raw_videos/        # Should contain downloaded videos
ls data/data_prepare/review_progress/   # Should contain clip_markings.json
```

### Step 3 Command Line Options

| Parameter | Description | Example |
|-----------|-------------|---------|
| `--status` | Show extraction status and statistics | `uv run s3_extract_clips.py --status` |
| `--force` | Re-extract existing clips (overwrite) | `uv run s3_extract_clips.py --force` |
| `--categories CAT1,CAT2` | Extract only specified categories | `uv run s3_extract_clips.py --categories accidents_general,road_rage` |
| `--output-dir DIR` | Custom output directory | `uv run s3_extract_clips.py --output-dir ./custom_clips` |

### Step 3 Output Structure

The extraction tool creates a well-organized final dataset:

```
data/dashcam/
├── 002_accidents_general_near_miss_6Iz-K5FszYM_0007.mp4
├── 003_accidents_general_collision_accident_6Iz-K5FszYM_0059.mp4
├── 004_road_rage_aggressive_behavior_ABC123XYZ_0045.mp4
├── extraction_metadata.json          # Comprehensive metadata
└── extraction_log.json              # Detailed extraction log
```

### Step 3 Filename Convention

**Format**: `{seq:03d}_{category}_{event_type}_{video_id}_{start_time}.mp4`

**Components**:
- **Sequence Number** (002, 003, 004...): Continues from existing clips, ensures uniqueness
- **Category**: From original download category (e.g., `accidents_general`, `road_rage`)
- **Event Type**: User-assigned event type from Step 2 (e.g., `near_miss`, `collision_accident`)
- **Video ID**: YouTube video ID for traceability
- **Start Time**: Clip start time (seconds, zero-padded)

**Examples**:
- `002_accidents_general_collision_accident_6Iz-K5FszYM_0059.mp4`
- `003_road_rage_aggressive_behavior_ABC123XYZ_0045.mp4`
- `004_intersections_near_miss_DEF456GHI_0123.mp4`

### Step 3 Key Features

#### ✅ **Perfect Timing Alignment**
- **Accurate seeking** using `ffmpeg -ss before -i` for precise timing
- **1.0000 similarity** between preview and extracted clips
- **Stream copying** preserves quality while maintaining speed

#### ✅ **Smart Resume Capability**  
- **Detects existing clips** and skips re-extraction by default
- **Sequence numbering** continues from highest existing clip number
- **Handles interruptions** gracefully - restart anytime
- **Force mode** available to re-extract if needed

#### ✅ **Comprehensive Metadata Preservation**
- **Combines data** from Step 1 (download metadata) and Step 2 (clip markings)
- **Preserves all information**: video metadata, clip timings, user annotations
- **Export format**: JSON compatible with evaluation pipeline
- **Statistics generation**: Categories, severities, durations, file sizes

### Advanced Usage Examples

#### Step 1 Advanced Examples
```bash
# Large dataset with custom directory
uv run evaluation/make_dataset/s1_youtube_downloader.py --total-videos 200 --output-dir /path/to/storage

# Resume with status check first
uv run evaluation/make_dataset/s1_youtube_downloader.py --status
uv run evaluation/make_dataset/s1_youtube_downloader.py --resume --total-videos 100

# Export metadata after download
uv run evaluation/make_dataset/s1_youtube_downloader.py --export-metadata dataset_info.json
```

#### Step 3 Advanced Examples  
```bash
# Extract specific categories only
uv run evaluation/make_dataset/s3_extract_clips.py --categories accidents_general,road_rage

# Extract intersection and merging scenarios
uv run evaluation/make_dataset/s3_extract_clips.py --categories intersections,highway_merging

# Extract to custom location
uv run evaluation/make_dataset/s3_extract_clips.py --output-dir /path/to/custom/clips

# Force re-extract with custom directory
uv run evaluation/make_dataset/s3_extract_clips.py --force --output-dir ./evaluation_clips
```

## 📊 Complete Workflow Output Structure

The 4-step pipeline creates a comprehensive, organized data structure:

```
data/
├── data_prepare/
│   ├── raw_videos/                        # Step 1: Downloaded videos by category
│   │   ├── .download_checkpoint.json      # Progress tracking & metadata
│   │   ├── video_metadata.json           # Exported metadata (optional)
│   │   ├── red_light_violations/
│   │   │   ├── abc123_Red_Light_Running.mp4
│   │   │   ├── abc123_Red_Light_Running.info.json
│   │   │   └── def456_Traffic_Violation.mp4
│   │   ├── road_rage/
│   │   │   ├── ghi789_Aggressive_Driver.mp4
│   │   │   └── jkl012_Road_Rage_Incident.mp4
│   │   ├── pedestrian_conflicts/
│   │   │   └── mno345_Crosswalk_Near_Miss.mp4
│   │   └── intersections/
│   │       └── pqr678_Intersection_Close_Call.mp4
│   └── review_progress/                   # Step 2: Clip markings and session data
│       ├── clip_markings.json            # Main clip markings file
│       ├── video_queue.json              # Review session queue
│       └── review_state.json             # Session state tracking
├── dashcam/                               # Step 3: Final curated dataset
│   ├── 002_accidents_general_near_miss_6Iz-K5FszYM_0007.mp4
│   ├── 003_accidents_general_collision_accident_6Iz-K5FszYM_0059.mp4
│   ├── extraction_metadata.json          # Comprehensive metadata
│   └── extraction_log.json              # Detailed extraction log
└── ground_truth/                          # Step 4: Ground truth annotations
    ├── {video_id}_ground_truth.json      # Individual ground truth files
    ├── exports/                          # Exported annotation files
    └── sessions/                         # Session and progress tracking
```

### File Structure Details

- **Video Files**: `{video_id}_{safe_title}.mp4` - Downloaded videos with unique IDs
- **Info Files**: `{video_id}_{safe_title}.info.json` - yt-dlp metadata for each video
- **Checkpoint**: `.download_checkpoint.json` - Progress tracking and comprehensive metadata
- **Metadata Export**: `video_metadata.json` - Optional export of all video information

## ⚙️ Configuration Options

### Video Categories

The system downloads videos from **30+ specialized categories** with **175+ targeted search queries**:

#### 🔴 **Critical Safety Violations**
- **red_light_violations** - Running red lights, intersection violations
- **stop_sign_violations** - Rolling stops, failure to stop completely
- **wrong_way_driving** - Head-on collision scenarios, highway wrong-way
- **school_bus_violations** - Illegal passing, child safety risks

#### ⚡ **Aggressive Driving Behaviors**
- **road_rage** - Confrontational driving, aggressive incidents
- **tailgating** - Following too close, brake checking
- **dangerous_overtaking** - Unsafe passing, oncoming traffic risks
- **cutting_off** - Unsafe lane changes, merge conflicts
- **speeding** - Excessive speed, street racing scenarios

#### 🚦 **Intersection & Traffic Control**
- **intersections** - Traffic light scenarios, yield failures
- **roundabouts** - Yield violations, wrong way entry
- **yield_violations** - Right-of-way failures, failure to yield

#### 🛣️ **Highway & Merging Scenarios**
- **highway_merging** - On-ramp failures, dangerous exits
- **highway_lane_changes** - Weaving, blind spot accidents
- **following_distance** - Rear-end collision scenarios

#### 👥 **Vulnerable Road Users**
- **pedestrian_conflicts** - Crosswalk violations, school zones
- **cyclist_conflicts** - Bike lane violations, dooring incidents
- **motorcycle_interactions** - Lane splitting, blind spot issues

#### 🌧️ **Environmental & Weather Hazards**
- **weather_rain** - Hydroplaning, wet road conditions
- **weather_snow_ice** - Winter driving, slippery conditions
- **weather_fog** - Low visibility conditions
- **night_driving** - Visibility issues, headlight problems

#### 🚧 **Construction & Work Zones**
- **construction_zones** - Worker safety, speed violations
- **emergency_vehicles** - Failure to yield, blocking access
- **vehicle_breakdowns** - Highway hazards, emergency positioning

#### 🚗 **Vehicle-Specific Scenarios**
- **truck_interactions** - Blind spot accidents, brake failures
- **bus_interactions** - Bus stop hazards, lane violations
- **parking_accidents** - Backing incidents, garage collisions

#### 🏗️ **Infrastructure & Special Conditions**
- **railroad_crossings** - Gate violations, train conflicts
- **bridge_tunnel** - Clearance issues, tunnel safety
- **road_debris** - Object hazards, avoidance maneuvers
- **animal_crossings** - Wildlife encounters, pet hazards

#### 🧠 **Driver Behavior Issues**
- **distracted_driving** - Phone use, driver inattention
- **drowsy_driving** - Fatigue-related incidents
- **lane_violations** - Improper lane changes, solid line crossing

#### 🏙️ **Urban & Complex Scenarios**
- **urban_complex** - City driving, congested conditions
- **accidents_general** - Comprehensive accident scenarios

## 📈 Performance Characteristics

### Complete Pipeline Performance

For typical workflow runs:

#### Step 1: Video Download
- **Download Time**: 15-45 minutes (depending on network speed and video count)
- **Video Count**: Configurable (e.g., 50 total videos distributed across 30+ categories)
- **Storage Requirements**: 2-8 GB (raw video files, ~50-150MB per video)
- **Resume Capability**: Instant resume without re-processing completed downloads

#### Step 2: Manual Review
- **Review Speed**: Varies by user (typically 2-5 minutes per video)
- **Clip Marking**: Real-time drag-and-drop interface
- **Session Management**: Automatic progress tracking and resume capability

#### Step 3: Clip Extraction
- **Processing Speed**: ~20-30 clips per minute (accurate seeking is slightly slower)
- **Timing Accuracy**: Perfect alignment with video reviewer preview (1.0000 similarity)
- **Storage Efficiency**: Lossless stream copying (no quality loss)
- **Resume Capability**: Instant resume from interruptions

### Advanced Features

#### Progress Tracking
- **Overall Progress Bar**: Shows total download progress across all categories
- **Category Progress**: Individual progress for each category being processed
- **Status Updates**: Real-time display of current video being downloaded
- **Completion Statistics**: Running totals of completed/failed/skipped downloads

#### Metadata Management
- **Comprehensive Storage**: Video details, file info, download timestamps
- **Export Formats**: JSON export for external analysis and integration
- **Search Integration**: Prevents duplicate downloads across multiple search sessions
- **File Verification**: Ensures downloaded files exist and are accessible

#### Quality Filtering

The downloader includes enhanced quality controls:
- **Duration Filtering** - Videos between 2-30 minutes (configurable)
- **Resolution Standards** - Preference for 720p MP4 format for consistency
- **Content Filtering** - Advanced dashcam-specific search terms and relevance scoring
- **Duplicate Prevention** - Multi-level duplicate detection by ID, URL, and content
- **Relevance Scoring** - Prioritizes videos with higher safety scenario relevance

## 🛠️ Development & Customization

### Adding New Search Categories

To add new video categories, modify `01_youtube_downloader.py`:

```python
# Add to the search_queries list in get_search_queries()
{"category": "new_category", "query": "dashcam search terms for new scenario"}
```

### Programmatic Usage

```python
# Add to path or run from appropriate directory
from evaluation.make_dataset.youtube_downloader import YouTubeDownloader
from pathlib import Path

# Initialize downloader with custom directory
downloader = YouTubeDownloader(Path("./custom_videos"))

# Check existing downloads
status = downloader.get_resume_status()
print(f"Already downloaded: {status['total_downloaded']} videos")

# Download with progress tracking
downloaded = downloader.download_dataset(
    total_videos=100,
    resume=True  # Resume if interrupted
)

# Export metadata for analysis
metadata_file = downloader.export_metadata(Path("analysis_metadata.json"))
print(f"Metadata exported to: {metadata_file}")
```

### Custom Configuration

```python
# Custom quality settings
downloader = YouTubeDownloader(download_dir)

# Override search queries for specific needs
downloader.search_queries = [
    {"category": "custom_scenario", "query": "dashcam custom search terms"},
    {"category": "specific_event", "query": "dashcam intersection near miss"}
]

# Download with custom parameters
results = downloader.download_dataset(total_videos=50)
```

### Metadata Structure

The exported metadata includes comprehensive information:

```json
{
  "export_time": "2025-01-15T10:30:00",
  "total_videos": 50,
  "download_directory": "/path/to/videos",
  "videos": {
    "category:video_id": {
      "video_id": "abc123xyz",
      "title": "Dashcam: Close Call at Red Light",
      "url": "https://youtube.com/watch?v=abc123xyz",
      "category": "red_light_violations",
      "duration": 180,
      "view_count": 15420,
      "upload_date": "20240315",
      "relevance_score": 4,
      "file_path": "/path/to/video.mp4",
      "file_size": 52428800,
      "download_time": "2025-01-15T10:25:30"
    }
  }
}
```

## 🔧 Troubleshooting

### Common Issues

**Dependencies Not Found**
```bash
# Install all required dependencies
uv sync  # Installs yt-dlp, tqdm, and other requirements

# Or install manually
pip install yt-dlp tqdm
```

**FFmpeg Not Found**
```bash
# Install FFmpeg
brew install ffmpeg  # macOS
apt install ffmpeg   # Linux
```

**Download Failures**
```bash
# Update yt-dlp to latest version
pip install --upgrade yt-dlp

# Check specific video
uv run evaluation/make_dataset/01_youtube_downloader.py --status
```

**Resume Issues**
```bash
# Check current status first
uv run evaluation/make_dataset/01_youtube_downloader.py --status

# Resume with exact same parameters as original run
uv run evaluation/make_dataset/01_youtube_downloader.py --resume --total-videos 50
```

**Progress Bar Issues**
- Ensure `tqdm` is installed: `pip install tqdm`
- Progress bars work best in terminals that support cursor positioning
- Use `--status` command to check progress without running downloads

**Checkpoint Corruption**
```bash
# If checkpoint file is corrupted, remove it to start fresh
rm data/data_prepare/raw_videos/.download_checkpoint.json

# Or check current status
uv run evaluation/make_dataset/01_youtube_downloader.py --status
```

### Debug Mode

Enable detailed logging by modifying the log level:
```python
import logging
logging.basicConfig(level=logging.DEBUG)
```

### Performance Tips

- **Large Downloads**: Use `--total-videos` with reasonable numbers (50-200)
- **Network Issues**: Videos will be retried automatically; use `--resume` after network recovery
- **Storage Space**: Monitor disk space; videos average 50-150MB each
- **Duplicate Prevention**: The system automatically prevents re-downloading; trust the skip messages

## 🛠️ Troubleshooting

### Step 3: Clip Extraction Issues

**No clips found to extract**
```bash
# Check if Step 2 was completed
ls data/data_prepare/review_progress/clip_markings.json

# Run video reviewer if file missing
cd evaluation/make_dataset/s2_video_reviewer/
uv run main.py
```

**FFmpeg not found**
```bash
# Install ffmpeg
brew install ffmpeg  # macOS
apt install ffmpeg   # Linux

# Verify installation
ffmpeg -version
```

**Source video files missing**
```bash
# Check video files exist
ls data/data_prepare/raw_videos/*/

# Re-run Step 1 if missing
uv run evaluation/make_dataset/s1_youtube_downloader.py --status
```

**Extraction failures**
```bash
# Check extraction log for details
cat data/dashcam/extraction_log.json

# Try with force mode
uv run evaluation/make_dataset/s3_extract_clips.py --force

# Check specific categories
uv run evaluation/make_dataset/s3_extract_clips.py --categories accidents_general
```

### Step 1: Download Issues

**Dependencies Not Found**
```bash
# Install all required dependencies
uv sync  # Installs yt-dlp, tqdm, and other requirements
```

**FFmpeg Not Found**
```bash
# Install FFmpeg
brew install ffmpeg  # macOS
apt install ffmpeg   # Linux
```

**Download Failures**
```bash
# Update yt-dlp to latest version
pip install --upgrade yt-dlp

# Check specific video
uv run evaluation/make_dataset/s1_youtube_downloader.py --status
```

## 📋 Usage with Manual Analysis

After downloading videos, you can proceed with manual analysis:

1. **Review Downloaded Videos**: Check `data/data_prepare/raw_videos/` for quality and relevance
2. **Manual Scene Analysis**: Analyze videos manually to identify specific safety scenarios
3. **Create Ground Truth**: Use findings to create evaluation ground truth annotations
4. **Organize Dataset**: Move selected clips to appropriate evaluation directories

## 🤝 Integration

## 🎯 Step 4: Ground Truth Annotation Tool (Detailed)

### Overview

The Ground Truth Annotation Tool provides a comprehensive web-based interface for creating detailed ground truth data for dashcam video clips. This tool implements a 5-step pipeline to generate complete annotations that can be used for AI model evaluation and training.

### Prerequisites for Step 4

```bash
# Ensure Step 3 has been completed with video clips available
ls data/dashcam/*.mp4  # Should contain extracted video clips

# Install any additional dependencies if needed
uv sync
```

### Step 4 Features

#### 🎬 **Professional Video Player**
- **Video.js Integration**: Professional-grade video playback with timeline controls
- **Frame-by-frame Navigation**: Precise video positioning for accurate annotation
- **Responsive Design**: Works on desktop, tablet, and mobile devices
- **Keyboard Shortcuts**: Efficient navigation (space bar, arrow keys, etc.)

#### 🔧 **5-Step Annotation Pipeline**

1. **Video Annotation**: Generate frame-by-frame descriptions of video content
   - Automatic annotation generation using AI models
   - Manual editing and optimization capabilities
   - User feedback integration for improved accuracy

2. **Scene Extraction**: Identify and extract atomic traffic scenarios
   - Scene boundary detection from video annotations
   - Manual scene editing and refinement
   - Scene categorization and tagging

3. **Violation Analysis**: Detect and categorize traffic rule violations
   - Rule-based violation detection
   - Severity assessment and categorization
   - Manual verification and editing

4. **Accident Analysis**: Assess potential accident scenarios and risks
   - Risk factor identification
   - Contributing factor analysis
   - Accident probability scoring

5. **Assessment Generation**: Create comprehensive driving safety evaluations
   - Overall safety scoring (0-100)
   - Detailed feedback and recommendations
   - Safety rating classification

#### 📊 **Progress Tracking & Session Management**
- **Visual Progress Indicators**: Track completion status for each clip and step
- **Session Persistence**: Maintain progress across browser sessions
- **Auto-save Detection**: Prevent data loss with automatic saving
- **Multi-clip Management**: Handle multiple video clips in a single session

### Running Step 4

```bash
# Navigate to the annotation tool directory
cd evaluation/make_dataset/s4_annotate_ground_truth/

# Start the web server
uv run python -m evaluation.make_dataset.s4_annotate_ground_truth.app

# Access the application
# - Main Interface: http://localhost:8000
# - API Documentation: http://localhost:8000/docs
# - Health Check: http://localhost:8000/api/health
```

### Step 4 API Endpoints

The tool provides a comprehensive REST API:

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/api/clips` | GET | List all available video clips |
| `/api/clips/{id}/stream` | GET | Stream video content |
| `/api/clips/{id}/step/{n}/generate` | POST | Generate step data |
| `/api/clips/{id}/step/{n}` | PUT | Save step data |
| `/api/clips/{id}/ground-truth` | GET | Get complete ground truth |
| `/api/clips/{id}/export` | POST | Export ground truth as JSON |
| `/api/session/current` | GET | Get current session info |

### Step 4 Output Format

Ground truth data is saved in JSON format:

```json
{
  "annotation": "Frame-by-frame video description...",
  "scenes": [
    {
      "id": "scene_001",
      "description": "Vehicle approaching intersection",
      "start_time": 5.2,
      "end_time": 12.8,
      "category": "intersection_approach"
    }
  ],
  "violations": [
    {
      "type": "red_light_violation",
      "severity": "high",
      "description": "Vehicle ran red light at intersection",
      "timestamp": 8.5
    }
  ],
  "accidents": [
    {
      "risk_level": "high",
      "scenario": "T-bone collision potential",
      "contributing_factors": ["red_light_violation", "excessive_speed"],
      "probability": 0.85
    }
  ],
  "assessment": {
    "overall_score": 25,
    "safety_rating": "dangerous",
    "feedback": "Critical safety violations detected...",
    "recommendations": ["Improve intersection awareness", "Follow traffic signals"]
  }
}
```

### With DriveGuard Evaluation

The complete 4-step pipeline integrates with the evaluation system:

```bash
# After ground truth annotation, clips are ready for evaluation
ls data/dashcam/*.mp4
ls data/ground_truth/*.json

# Run evaluation on the curated dataset with ground truth
uv run evaluation/1_prepare_evaluation_data.py
uv run evaluation/3_run_ragas_evaluation.py
```

## 🚀 Step 5: System Output Generation (Detailed)

### Overview

The System Output Generation tool provides automated processing for creating system outputs that can be compared against ground truth annotations. This tool supports multiple AI models and components for comprehensive evaluation.

### Prerequisites for Step 5

```bash
# Ensure Step 4 has been completed with ground truth annotations
ls data/evaluation/ground_truth/*.json  # Should contain ground truth files

# Ensure model configuration files are available
ls evaluation/models/text.txt           # For scene, violation, accident, assessment
ls evaluation/models/annotation.txt     # For annotation component

# Install dependencies and ensure API keys are configured
uv sync
# Configure API keys in config/.env
```

### Step 5 Features

#### 🤖 **Multi-Model Support**
- **23 supported models** including GPT-4o, GPT-4.1, Claude Opus, Gemini, Groq models, and more
- **Gateway integration** for custom model routing and cost optimization
- **Automatic model validation** and configuration checking
- **Batch processing** with intelligent error handling and retries

#### 📋 **Component Pipeline**
1. **Annotation**: Generate video frame-by-frame descriptions
2. **Scene**: Extract atomic traffic scenarios from annotations
3. **Violation**: Detect traffic rule violations using Milvus database
4. **Accident**: Analyze accident risks using Neo4j graph database
5. **Assessment**: Generate comprehensive driving safety evaluations

#### ⏱️ **Performance Tracking**
- **Runtime monitoring** for each component and model
- **Processing speed** metrics and bottleneck identification
- **Individual scene timing** for multi-scene processing
- **Session tracking** across multiple model runs

#### 🔄 **Resume Capability**
- **Skip-if-exists logic** to avoid reprocessing completed outputs
- **Intelligent checkpoint system** for interruption recovery
- **File integrity validation** before skipping existing outputs
- **Model tracking** with detailed run history

### Running Step 5

#### Basic Commands

```bash
# Navigate to the system output generation directory
cd evaluation/make_dataset/s5_generate_system_output/

# Generate outputs for all models in text.txt (scene, violation, accident, assessment)
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --component scene
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --component violation

# Generate outputs for all models in annotation.txt (annotation component only)
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --component annotation

# Generate for a specific model only
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --component violation --model "openai:gpt-4o"
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --component scene --model "groq:llama-3.3-70b-versatile"

# Process specific videos only
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --component violation --videos "1,2,3"

# Process all videos (including those without ground truth)
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --component scene --all

# Force overwrite existing outputs
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --component violation --overwrite
```

#### Status and Monitoring

```bash
# Check overall status of all system outputs
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --status

# Check detailed status for specific component and model
uv run python -m evaluation.make_dataset.s5_generate_system_output.main --status --component violation --model "openai:gpt-4o"
```

### Step 5 Command Line Options

| Parameter | Description | Example |
|-----------|-------------|---------|
| `--component COMP` | Component to process | `--component violation` |
| `--model MODEL` | Specific model ID | `--model "openai:gpt-4o"` |
| `--videos IDs` | Comma-separated video IDs | `--videos "1,2,3"` |
| `--status` | Show processing status | `--status` |
| `--overwrite` | Force overwrite existing outputs | `--overwrite` |
| `--all` | Process all videos | `--all` |

### Step 5 Output Structure

Generated outputs are organized by component and model:

```
data/evaluation/system_outputs/
├── annotation/                    # Video annotation outputs
│   ├── gpt-4o/
│   │   ├── 0000_cut_off_accident.json
│   │   └── 0001_left_turn_cut_off.json
│   ├── claude-opus-4-1-20250805/
│   └── gemini-2.5-pro/
├── scene/                         # Scene extraction outputs  
│   ├── gpt-4o/
│   ├── llama-3.3-70b-versatile/
│   └── deepseek-r1-distill-llama-70b/
├── violation/                     # Traffic violation detection outputs
│   ├── gpt-4o/
│   │   ├── 0000_cut_off_accident.json    # Contains multiple scene violations
│   │   └── 0001_left_turn_cut_off.json
│   └── gpt-4.1/
├── accident/                      # Accident analysis outputs
└── assessment/                    # Final assessment outputs
```

### Step 5 Performance Characteristics

#### Multi-Model Processing Performance
- **Scene Extraction**: ~5-15 seconds per ground truth file (varies by model)
- **Violation Detection**: ~10-30 seconds per file (includes Milvus DB queries)
- **Concurrent Processing**: Models processed sequentially with full progress tracking
- **Error Handling**: Automatic retry logic with graceful degradation

#### Runtime Tracking Example
```json
{
  "total_processing_time": 10.90,
  "scenes_processed": 5,
  "average_time_per_scene": 2.18,
  "violations_detected": 3,
  "violations_per_scene": [
    {"scene": 1, "violation": true, "time": 2.45},
    {"scene": 2, "violation": false, "time": 1.89},
    {"scene": 3, "violation": true, "time": 2.12},
    {"scene": 4, "violation": true, "time": 2.31},
    {"scene": 5, "violation": false, "time": 2.13}
  ]
}
```

### Pipeline Compatibility

- **Step 1 Output**: Raw videos in `data/data_prepare/raw_videos/` by category
- **Step 2 Output**: Clip markings in `data/data_prepare/review_progress/clip_markings.json`
- **Step 3 Output**: Final clips in `data/dashcam/` as expected by evaluation scripts
- **Step 4 Output**: Ground truth annotations in `data/evaluation/ground_truth/` in JSON format
- **Step 5 Output**: System outputs in `data/evaluation/system_outputs/` organized by component and model
- **Format**: MP4 files and JSON annotations compatible with evaluation pipeline
- **Metadata**: Comprehensive metadata available for analysis and ground truth validation
- **Naming**: Sequential naming ensures consistent processing order

### File Organization

```
data/
├── data_prepare/           # Intermediate data (raw videos, review progress)
│   ├── raw_videos/        # Step 1: Downloaded videos by category
│   └── review_progress/   # Step 2: Clip markings and session data
├── dashcam/               # Step 3: Final curated dataset
│   ├── *.mp4             # Extracted clips with meaningful names
│   ├── extraction_metadata.json
│   └── extraction_log.json
├── ground_truth/          # Step 4: Ground truth annotations
│   ├── *_ground_truth.json  # Individual annotation files
│   ├── exports/            # Exported ground truth data
│   └── sessions/           # Progress and session tracking
└── evaluation/            # Step 5: System output generation
    ├── system_outputs/    # Generated system outputs by component and model
    │   ├── annotation/    # Video annotation outputs
    │   ├── scene/         # Scene extraction outputs  
    │   ├── violation/     # Traffic violation detection outputs
    │   ├── accident/      # Accident analysis outputs
    │   └── assessment/    # Final assessment outputs
    └── tracking/          # Performance tracking and runtime data
```

## 🔒 Legal & Ethical Considerations

- **YouTube Terms** - Respect YouTube's terms of service for video downloads
- **Privacy** - Downloaded videos are public dashcam footage
- **Fair Use** - Educational and research purposes only  
- **Data Protection** - No personal information is collected or stored

## 📞 Support

For issues with the YouTube downloader:
1. Check this README for common solutions
2. Verify yt-dlp installation and version
3. Test with a single category first
4. Check network connectivity and YouTube accessibility

---

## 📊 Progress Tracking Example

When running the downloader, you'll see progress like this:

```
YOUTUBE DASHCAM VIDEO DOWNLOADER
============================================================

Target: 50 videos total (distributed across categories)

Video distribution preview:
----------------------------------------
accidents_general: 2 videos
animal_crossings: 2 videos
bridge_tunnel: 2 videos
[... more categories ...]

Starting download process...
Overall Progress: 45%|████████████████████████████████████████████████████████                                                  | 23/50 [15:30<12:45, 2.1it/s]
red_light_violations: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [02:15<00:00, 1.2it/s]

Download Summary:
----------------------------------------
accidents_general               :   2 videos
animal_crossings               :   2 videos
[... complete summary ...]

Total downloaded: 47/50 videos
```

**Last Updated**: January 2025  
**Version**: 3.0  
**Compatibility**: DriveGuard v1.0+, Python 3.12+, yt-dlp 2023.1+, tqdm 4.0+, FastAPI 0.100+