# GDM Dangerous Capabilities: In House CTF

This directory includes 13 Capture The Flag challenges from Google DeepMind's [Dangerous Capabilities Evaluations](https://github.com/google-deepmind/dangerous-capability-evaluations/tree/main/in_house_ctf) (originally published in [Evaluating Frontier Models for Dangerous Capabilities](https://arxiv.org/pdf/2403.13793)).

<!-- Contributors: Automatically Generated -->
Contributed by [@XkunW](https://github.com/XkunW)
<!-- /Contributors: Automatically Generated -->

<!-- Usage: Automatically Generated -->
## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Or, if developing on a clone of the `inspect_evals` repo, you can install the package in editable mode with:

```bash
pip install -e ".[dev]"
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/gdm_in_house_ctf --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```
<!-- /Usage: Automatically Generated -->

>[!NOTE]
>This implementation uses [tool calling](https://inspect.ai-safety-institute.org.uk/tools.html) to enable the model to execute bash commands. Note that the bash commands are executed inside Docker containers, so you will need to install [Docker Engine](https://docs.docker.com/engine/install/) in order to run the evaluation.
>
>Each task runs using the task-specific `compose.yaml` files, which pulls the remote Docker images for the attacker and victim environments. See the [Docker Images](#docker-images) section below for details on using locally built Docker images.


<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/gdm_in_house_ctf --limit 10
inspect eval inspect_evals/gdm_in_house_ctf --max-connections 10
inspect eval inspect_evals/gdm_in_house_ctf --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->


To evaluate one or more individual challenges use a task parameter:

``` bash
inspect eval inspect_evals/gdm_in_house_ctf -T challenges=grafana,db_3 
```

You can also evaluate multiple models at once:

``` bash
inspect eval inspect_evals/gdm_in_house_ctf --model openai/gpt-4o,google/gemini-1.5-pro
```



### Agent

The challenges are by default evaluated using a basic ReAct agent that implements a tool use loop with encouragement for the model to keep going if its gives up. The default agent will use up to 50 messages to solve the challenge, you can customise this using the `max_messages` task option:

For example:

``` bash
inspect eval inspect_evals/gdm_in_house_ctf -T max_messages=75
```

You can also run the task with an entirely different solver agent suing the `--solver` option. For example, if you have defined a solver agent in the source file `myagent.py`:

```bash
inspect eval inspect_evals/gdm_in_house_ctf --solver myagent.py 
```

### Epochs

The GDM paper evaluated each challenge 10 times, reporting success if at least 1 challenge of the 10 was successful. The same default number of epochs (10) is used here, however you can modify this with the `--epochs` option:

``` bash
inspect eval inspect_evals/gdm_in_house_ctf --epochs=5
```

## Docker Images

Each task runs using the task-specific `compose.yaml` files, which pulls the remote Docker images for the attacker and victim environments. The Dockerfiles used to build these images have been provided as well (adapted from the original repository with minor modifications).

Note that the flag values in the Dockerfiles have been replaced with placeholders `REPLACE_ME_WITH_RANDOM_FLAG_N` to prevent them from ending up in web crawler accessible GitHub data. The actual values are baked into the remote Docker images and will be accessed during evaluation run. If you'd like to build images locally images, make sure to replace these placeholders with your actual flag values and update the `compose.yaml` files to use local docker images. For example, you would replace this:

``` yaml
services:
  idor:
    image: marshw/idor
```

With this:

``` yaml
services:
  default:
    build: .
```

Again, remembering to replace the `REPLACE_ME_WITH_RANDOM_FLAG_N` within the Dockerfile.

## Scoring
A simple accuracy is calculated over the datapoints.
