# Cybench

[Cybench](https://cybench.github.io/) is a benchmark for evaluating models' cybersecurity capabilities. It contains 40 capture the flag (CTF) tasks from 4 CTF competitions that cover 6 domains: cryptography, web security, reverse engineering, forensics, exploitation (pwn), and miscellaneous.

<!-- Contributors: Automatically Generated -->
Contributed by [@sinman-aisi](https://github.com/sinman-aisi), [@sam-deverett-dsit](https://github.com/sam-deverett-dsit), [@kola-aisi](https://github.com/kola-aisi), [@pgiav](https://github.com/pgiav)
<!-- /Contributors: Automatically Generated -->

<!-- Usage: Automatically Generated -->
## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Or, if developing on a clone of the `inspect_evals` repo, you can install the package in editable mode with:

```bash
pip install -e ".[dev]"
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/cybench --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```
<!-- /Usage: Automatically Generated -->

>[!NOTE]
>The bash and python code is executed inside a Docker container, so you will need to install [Docker Engine](https://docs.docker.com/engine/install/) in order to run the evaluation.
>
> Docker containers are also used in some challenges for auxiliary services. When you run this, the containers are pulled from a public registry. The source code is still included, however, both for your reference and because some files are copied from them into the agent's container at runtime.

<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/cybench --limit 10
inspect eval inspect_evals/cybench --max-connections 10
inspect eval inspect_evals/cybench --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->

You can specify a subset of challenges to run. For example:
```bash
inspect eval inspect_evals/cybench -T challenges=avatar,back_to_the_past
```

You can specify a certain variant to run. For example, to run the solution variant for each challenge (that has one), you can run:
```bash
inspect eval inspect_evals/cybench -T variants=solution
```

You can also create and specify an agent to use so long as it's in the form of an Inspect [solver](https://inspect.ai-safety-institute.org.uk/solvers.html). See `default_agent` in [cybench.py](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/cybench/cybench.py) for an example.

There are two task parameters that define limits on the evaluation:
- `max_attempts` defines the number of incorrect submissions to allow before ending the challenges (defaults to 3).
- `max_messages` defines the maximum number of messages allowed in the conversation before ending the challenges (defaults to 30).

For example:
```bash
inspect eval inspect_evals/cybench -T max_attempts=5 -T max_messages=75
```

## Scoring
A simple proportion is calculated over the challenges.
