# Standalone Multi-Model Analysis Suite

This directory contains a self-contained, reproducible analysis package for comparing multi-agent simulation results across different language models. The package includes all necessary code, dependencies, and automation scripts to generate comprehensive analysis reports.

## Quick Start

To run the complete analysis, simply execute the bootstrap script from this directory:

```bash
./reproduce_all.sh
```

This single command will:
1. Create a Python virtual environment
2. Install all required dependencies
3. Run the complete analysis pipeline
4. Generate timestamped reports in `analysis_report_YYYYMMDD_HHMMSS/`

## What You Get

Each analysis run produces:

### Main Analysis Report (`analysis_report.html`)
- **Key Survival Metrics**: Overall performance statistics across all models
- **Data Quality Metrics**: Validation rates and data integrity statistics
- **Survival Analysis by Factors**: Kaplan-Meier curves, hazard rates, and event distributions
- **Ablation Analysis**: Factor-specific survival comparisons
- **Behavioral Analysis**: Tool usage patterns and social influence effects
- **Statistical Model Results**: Discrete-time hazard model with forest plot

### Suite Comparison Report (`suite_comparison_report.html`)
- **Cross-Model Survival Plot**: Direct comparison of all models on a single chart
- **Social Influence Overlay**: Broadcast (solid) vs. Isolated (dashed) conditions

### Supporting Files
- `summary_metrics.json`: Machine-readable key statistics
- `model_results.json`: Statistical model outputs
- `*.png`: All publication-ready plots

## Directory Structure

```
all_models/
├── reproduce_all.sh                    # Main execution script
├── scripts/
│   ├── analyze_standalone.py           # Main analysis script
│   └── analysis_plots/                 # Local plotting modules
│       ├── kaplan_meier_plotter.py
│       ├── hazard_plotter.py
│       ├── ablation_plotter.py
│       └── ... (other plotting modules)
├── openrouter-model-name-1/            # Experiment folders
│   └── aggregated_data/
│       ├── agent_outcomes.csv
│       ├── step_level_data.csv
│       ├── cell_aggregates.csv
│       └── cell_summary.csv
├── openrouter-model-name-2/
│   └── aggregated_data/
│       └── ... (same CSV structure)
└── analysis_report_YYYYMMDD_HHMMSS/    # Generated reports (timestamped)
    ├── analysis_report.html
    ├── suite_comparison_report.html
    ├── summary_metrics.json
    ├── model_results.json
    └── *.png (all plots)
```

## Manual Usage

If you need more control over the analysis, you can run the standalone script directly:

```bash
# First, set up the environment (only needed once)
python3 -m venv venv
source venv/bin/activate
pip install pandas numpy matplotlib seaborn statsmodels lifelines Jinja2

# Then run the analysis with custom parameters
python scripts/analyze_standalone.py \
    --suite_directory ./ \
    --output_directory ./my_custom_report \
    --survival-prob-range 0.3 1.0

# Clean up
deactivate
```

### Command-Line Options

- `--suite_directory`: Path to directory containing experiment folders (use `./` for current directory)
- `--output_directory`: Where to save the analysis report
- `--survival-prob-range`: Y-axis range for survival plots (e.g., `0.4 1.0` to focus on high survival rates)

## Requirements

- **Python 3.7+**: The analysis uses modern Python features
- **System Requirements**: ~2GB RAM for processing large datasets
- **Dependencies**: Automatically installed by `reproduce_all.sh`
  - pandas, numpy (data processing)
  - matplotlib, seaborn (visualization)
  - statsmodels (statistical modeling)
  - lifelines (survival analysis)
  - Jinja2 (report generation)

## Data Format

The analysis expects each experiment folder to contain an `aggregated_data/` subdirectory with these CSV files:
- `agent_outcomes.csv`: Individual agent results and survival data
- `step_level_data.csv`: Agent×step trajectories and behavioral data
- `cell_aggregates.csv`: Cell×step survival analysis data
- `cell_summary.csv`: Per-cell summary statistics

These files are typically generated by the main framework's `aggregate_data.py` script.

### Common Issues

**Missing aggregated_data folders**: Ensure each experiment directory contains the required CSV files in an `aggregated_data/` subdirectory.

**Permission errors**: Make sure `reproduce_all.sh` is executable:
```bash
chmod +x reproduce_all.sh
```

**Memory issues**: For very large datasets, you may need to increase available RAM or run the analysis on a more powerful machine.

**Virtual environment conflicts**: If you encounter package conflicts, delete the `venv/` directory and re-run `reproduce_all.sh`.

### Getting Help

This package is designed to be self-contained and work out of the box. If you encounter issues:

1. Check that all experiment folders contain the required `aggregated_data/` structure
2. Verify Python 3.7+ is installed on your system
3. Ensure sufficient disk space for the generated reports and plots
4. Review the console output for specific error messages
