# MMLU

[MMLU](https://arxiv.org/pdf/2009.03300) is a benchmark to measure the model's multitask accuracy. This dataset covers 57 tasks such as elementary mathematics, US history, computer science, law, and more.

There are also two tasks you can run here:
- 0-shot, Simple Evals style prompt (default, recommended for instruction-tuned models)
- 5-shot, traditional MMLU paper prompt (recommended for base models)

<!-- Contributors: Automatically Generated -->
Contributed by [@jjallaire](https://github.com/jjallaire), [@domdomegg](https://github.com/domdomegg)
<!-- /Contributors: Automatically Generated -->

<!-- Usage: Automatically Generated -->
## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Or, if developing on a clone of the `inspect_evals` repo, you can install the package in editable mode with:

```bash
pip install -e ".[dev]"
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/mmlu_0_shot --model openai/gpt-4o
inspect eval inspect_evals/mmlu_5_shot --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```
<!-- /Usage: Automatically Generated -->

<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/mmlu_0_shot --limit 10
inspect eval inspect_evals/mmlu_5_shot --max-connections 10
inspect eval inspect_evals/mmlu_0_shot --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->

You can also filter for specific MMLU subjects:

```bash
inspect eval inspect_evals/mmlu_0_shot -T subjects=anatomy
inspect eval inspect_evals/mmlu_5_shot -T subjects=anatomy,astronomy
```

## 0-shot Dataset

Here is an example prompt from the dataset (after it has been further processed by Inspect):

>Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of A,B,C,D.
>
>The constellation ... is a bright W-shaped constellation in the northern sky.
>
>A) Centaurus
>B) Cygnus
>C) Cassiopeia
>D) Cepheus

The model is then tasked to pick the correct choice.

## 5-shot Dataset

This prompt format is the same as the MMLU paper.
It uses 5 examples of correctly answered questions in the subject (from the `dev` set) before posing the test question:

> The following are multiple choice questions about computer security.
> 
> SHA-1 has a message digest of
> (A) 160 bits
> (B) 512 bits
> (C) 628 bits
> (D) 820 bits
> Answer: A
> 
> _____________ can modify data on your system – so that your system doesn’t run correctly or you can no longer access specific data, or it may even ask for ransom in order to give your access.
> (A) IM – Trojans
> (B) Backdoor Trojans
> (C) Trojan-Downloader
> (D) Ransom Trojan
> Answer: D
> 
> What is ethical hacking?
> (A) "Hacking" ethics so they justify unintended selfish behavior
> (B) Hacking systems (e.g., during penetration testing) to expose vulnerabilities so they can be fixed, rather than exploited
> (C) Hacking into systems run by those whose ethics you disagree with
> (D) A slang term for rapid software development, e.g., as part of hackathons
> Answer: B
> 
> Exploitation of the Heartbleed bug permits
> (A) overwriting cryptographic keys in memory
> (B) a kind of code injection
> (C) a read outside bounds of a buffer
> (D) a format string attack
> Answer: C
> 
> The ____________ is anything which your search engine cannot search.
> (A) Haunted web
> (B) World Wide Web
> (C) Surface web
> (D) Deep Web
> Answer: D
> 
> A packet filter firewall filters at the
> (A) Application or transport
> (B) Data link layer
> (C) Physical Layer
> (D) Network or transport layer
> Answer: 

We recommend using pretrained models over instruction-tuned models for this evaluation.
The MMLU prompt format is more of a 'fill in the blank' task which better suits pretrained models.
Some instruction-tuned models can get confused and try to talk through the problem instead.
For example, to evaluate Llama 3.1 we recommend `llama3.1:8b-text-q4_0` over `llama3.1:8b-instruct-q4_0`.

The one difference from the paper is that we take the answer directly given by the model.
The paper instead takes the logprobs for the tokens 'A', 'B', 'C' and 'D' and picks the greatest one.
We've made this change because getting logprobs is not possible with many models supported by Inspect.
This results in slightly lower scores than the paper's method, as models can output invalid tokens which are marked incorrect.

## Scoring

A simple accuracy is calculated over the datapoints.
