# CyberMetric

[CyberMetric](https://arxiv.org/abs/2402.07688) is a collection of four datasets containing 80, 500, 2000 and 10000 multiple-choice questions, designed to evaluate understanding across nine domains within cybersecurity: Disaster Recovery and BCP, Identity and Access Management (IAM), IoT Security, Cryptography, Wireless Security, Network Security, Cloud Security, Penetration Testing, and Compliance/Audit.

Contributed by [@neilshaabi](https://github.com/neilshaabi)

## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:
```bash
pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Then, evaluate against one or more models with:
```bash
inspect eval inspect_evals/cybermetric_80 --model openai/gpt-4o
inspect eval inspect_evals/cybermetric_500 --model openai/gpt-4o
inspect eval inspect_evals/cybermetric_2000 --model openai/gpt-4o
inspect eval inspect_evals/cybermetric_10000 --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```

## Options

You can control a variety of options from the command line. For example:
```bash
inspect eval inspect_evals/cybermetric_80 --limit 10
inspect eval inspect_evals/cybermetric_500 --max-connections 10
inspect eval inspect_evals/cybermetric_2000 --temperature 0.5
```

See `inspect eval --help` for all available options.

## Dataset

Here is an example prompt from the dataset (after it has been further processed by Inspect):

>Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of A,B,C,D.
>
>In cryptography, what is the purpose of using a key-derivation function (KDF)?
>
>
>A) Generate public keys, B) Authenticate digital signatures, C) Encrypt data using a password, D) KDF are algorithms used to transform a secret into crucial parameters like keys and Initialization Vectors (IVs).

The model is then tasked to pick the correct choice.

## Scoring

A simple accuracy is calculated over the datapoints.
