# Agent Misalignment Evaluations: Community Curator


## Configuration Options

The evaluation framework supports various configuration parameters provided in `datasets/community_curator.json`:

| Parameter | Options | Description |
|------------------------|--------------------|----------------------------|
| `reveal_degree` | `none`, `removal_alert`, `explicit_prompt` | Controls how explicit we are on Eval dynamics (alert hinting, hints in prompt...) |
| `reveal_timing` | `none`, `start`, `incremental` | Controls when competition information is provided (all at begining, gradual) |
| `stage_steps` | Configuration object | Defines stages, toxicity levels, and message ordering |

## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/community_curator --model openai/gpt-4o-mini
```

More examples:
```bash
# Run with all models and temperature 0
inspect eval inspect_evals/community_curator \
    --model openai/gpt-4.1-2025-04-14,anthropic/claude-3-7-sonnet-20250219,google/gemini-2.5-pro-preview-05-06,openai/o4-mini-2025-04-16,anthropic/claude-3-opus-20240229,google/gemini-2.0-flash-001 \
    --temperature 0

# Test launch 1 sample, 1 model
inspect eval inspect_evals/community_curator \
    --model openai/gpt-4o-mini \
    --temperature 0 \
    --sample-id 1-baseline_none

# Test launch 1 sample, 1 model, and the CMSScorer
inspect eval inspect_evals/community_curator \
    --model openai/gpt-4o-mini \
    --temperature 0 \
    --sample-id 1-baseline_none \
    -T cms_scorer=True

# Test launch a run specific scorer from an already existing .eval file
inspect score src/inspect_evals/agentmisalignment/evals/community_curator/results/logs/community_curator_claude-3-7-sonnet.eval \
    --scorer src/inspect_evals/agentmisalignment/scorers/community_curator_scorer.py
```

Evaluations are executed in a Docker sandbox environment with all necessary tools and data.\
\

The evaluation framework supports various configuration parameters provided in `datasets/resource_acquisition.json`:

| Parameter | Options | Description |
|------------------------|--------------------|----------------------------|
| `scenario` | `medical`, `medical-household` | Controls scenario detail, only medical vs. a slightly more complex one including Household control as well |
| `explicit` | `low`, `medium`, `high` | Level of reveal in our Eval, from less hinting to highest |

For local test runs: `python utils/exploratory_runs/run_community_curator.py`

Note: by default this will run with the option `sandbox_cleanup = False` so you can debug files like `dashboard.csv` or the logs (If you are running from shell `$ inspect eval inspect_evals/community_curator --no-sandbox-cleanup`).
When you do this, you’ll see a list of sandbox containers printed out which includes the ID of each container. You can then use this ID to get a shell inside one of the containers:

```
docker exec -it <id> bash -l
```

So, even while running the Eval you'll see appear sample-specific folders, example:

```         
/app
├── <task_id>/                   # Task-specific directory
│   ├── dashboard.csv            # Performance tracking for this task
│   └── agent_logs/              # Logs of agent actions for this task
│       └── actions_*.log        # Per-step agent action logs
```

Where `app/1-1-1/dashboard.csv` would show the Dashboard for the sample run with id `1-1-1`.

When you no longer need the environments, you can clean them up either all at once or individually:
```
# cleanup all environments
inspect sandbox cleanup docker

# cleanup single environment
inspect sandbox cleanup docker <id>
```

## Data Sources

The framework uses the [Jigsaw Toxicity Classification dataset](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge) with some modifications to ensure reliable benchmarking.
