# MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

[MMLU-Pro](https://arxiv.org/pdf/2406.01574) is a more robust and challenging massive multi-task understanding dataset tailored to more rigorously benchmark large language models' capabilities. This dataset contains 12K complex questions across 14 disciplines (including 'Other').

There are three major differences compared to original MMLU:

1. MMLU-Pro increases the number of options from 4 to 10, making the evaluation more realistic and challenging.
2. In this dataset, the creators increase the problem difficulty and integrate more reasoning-focused problems.
3. The benchmark is made more robust by increasing the number of distractor options.

<!-- Contributors: Automatically Generated -->
Contributed by [@xeon27](https://github.com/xeon27)
<!-- /Contributors: Automatically Generated -->

<!-- Usage: Automatically Generated -->
## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Or, if developing on a clone of the `inspect_evals` repo, you can install the package in editable mode with:

```bash
pip install -e ".[dev]"
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/mmlu_pro --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```
<!-- /Usage: Automatically Generated -->

<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/mmlu_pro --limit 10
inspect eval inspect_evals/mmlu_pro --max-connections 10
inspect eval inspect_evals/mmlu_pro --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->

## Dataset
Here is an example from the dataset:

>Question: Approximately how far away is the Andromeda Galaxy?
>
>Options:
>A) 5 million light years  
>B) 2.5 million light years  
>C) 2.1 million light years  
>D) 1.9 million light years  
>E) 3.2 million light years  
>F) 4 million light years  
>G) 1.7 million light years  
>H) 3.5 million light years  
>I) 1.2 million light years  
>J) 2.8 million light years  

The model is tasked to answer the question and choose the appropriate option.

## Evaluation
The prompts are based on EleutherAI's [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/mmlu_pro) and  [MultipleChoiceTemplate.SINGLE_ANSWER](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/solver/_multiple_choice.py). The in-built `choice` scorer is used for evaluation.
