# MATH: Measuring Mathematical Problem Solving

[MATH](https://arxiv.org/abs/2103.03874) is a dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. It consists of 5 levels of difficulty and 7 subjects.

The zero-shot prompt template is based on OpenAI's [simple-evals](https://github.com/openai/simple-evals/blob/main/math_eval.py) and the format of the few-shot examples is taken from https://arxiv.org/pdf/2206.14858.

## Takedown notice
As of 16 Jan 2025, the [MATH dataset](https://huggingface.co/datasets/hendrycks/competition_math) has been [taken down](https://huggingface.co/datasets/hendrycks/competition_math/discussions/5) from HuggingFace due to a [DMCA notice](https://huggingface.co/datasets/huggingface-legal/takedown-notices/blob/main/2025/2025-01-02-AoPS.md) from [The Art of Problem Solving](https://artofproblemsolving.com/).

<!-- Contributors: Automatically Generated -->
Contributed by [@xeon27](https://github.com/xeon27)
<!-- /Contributors: Automatically Generated -->

<!-- Usage: Automatically Generated -->
## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install "inspect_evals[math]@git+https://github.com/UKGovernmentBEIS/inspect_evals"
```

Or, if developing on a clone of the `inspect_evals` repo, you can install the package in editable mode with:

```bash
pip install -e ".[dev,math]"
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/math --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```
<!-- /Usage: Automatically Generated -->

<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/math --limit 10
inspect eval inspect_evals/math --max-connections 10
inspect eval inspect_evals/math --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->

## Dataset
Here is an example from the dataset:

>Problem: How many vertical asymptotes does the graph of $$y=\\frac{2}{x^2+x-6}$$ have? <br>
>Given Solution: The denominator of the rational function factors into $$x^2+x-6=(x-2)(x+3)$$. Since the numerator is always nonzero, there is a vertical asymptote whenever the denominator is $$0$$, which occurs for $$x = 2$$ and $$x = -3$$.  Therefore, the graph has $$\\boxed{2}$$ vertical asymptotes.

The model is tasked to solve the problem step by step and return the final answer.

## Scoring

Three scoring strategies are used:
1. ```expression_equivalance```: A grader model is used to compare the predicted mathematical answer/expression with the target.
2. ```expression_exact_match_sympy```: The answer and target are evaluated for exact match using the ```sympy``` package used in https://arxiv.org/pdf/2206.14858. This implementation is based on EleutherAI's [lm-evaluation-harness/minerva_math](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/minerva_math/utils.py#L144)
3. ```expression_exact_match```: The answer and target are evaluated for exact match using simple rules, based on EleutherAI's [lm-evaluation-harness/hendrycks_math](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/hendrycks_math.py#L88)
