# InterCode CTF

This directory includes an implementation of the [InterCode CTF](https://intercode-benchmark.github.io/#ctf) benchmark (originally published in [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://arxiv.org/abs/2306.14898)). Here we evaluate 81 of the 100 challenges (excluding the 19 challenges that require internet access).

<!-- Contributors: Automatically Generated -->
Contributed by [@jjallaire](https://github.com/jjallaire)
<!-- /Contributors: Automatically Generated -->

<!-- Usage: Automatically Generated -->
## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Or, if developing on a clone of the `inspect_evals` repo, you can install the package in editable mode with:

```bash
pip install -e ".[dev]"
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/gdm_intercode_ctf --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```
<!-- /Usage: Automatically Generated -->

>[!NOTE]
>This implementation uses [tool calling](https://inspect.ai-safety-institute.org.uk/tools.html) to enable the model to execute bash and python code. The system prompt from the paper has been modified accordingly, and also asks the model to reason step by step about each action it takes (this strategy is also [explored](https://github.com/princeton-nlp/intercode/blob/master/experiments/eval_react.py) in the paper).
>
>Note that bash and python code is executed inside a Docker container built from this [Dockerfile](Dockerfile.template), so you will need to install [Docker Engine](https://docs.docker.com/engine/install/) in order to run the evaluation. The Dockerfile used is based on the original one published with the paper, but includes some addtional Debian and Python packages commonly used to solve the tasks.


<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/gdm_intercode_ctf --limit 10
inspect eval inspect_evals/gdm_intercode_ctf --max-connections 10
inspect eval inspect_evals/gdm_intercode_ctf --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->

If you are experimenting, you might find it useful to evaluate only a random subset of the samples using the `shuffle` parameter and `--limit`:

```bash
$ inspect eval inspect_evals/gdm_intercode_ctf \
-T shuffle=true --limit 10 \
--model openai/gpt-4-turbo
```

There are two task parameters that define limits on the evaluation:

- `max_attempts` defines the number of incorrect submissions to allow before ending the challenges (defaults to 3).
- `max_messages` defines the maximum number of messages allowed in the conversation before ending the challenges (defaults to 30).

For example:

```bash
$ inspect eval inspect_evals/gdm_intercode_ctf \
-T max_attempts=5 -T max_messages=75 \
--model openai/gpt-4o
```

## Scoring
A simple accuracy is calculated over the datapoints.
