# MathVista: Evaluating Mathematical Reasoning in Visual Contexts

[MathVista](https://arxiv.org/pdf/2310.02255) to evaluate model performance in answering mathematics problems that have a visual component. Each problem has an associated image which is required to solve the problem. The dataset is made up of 6,141 questions from 28 existing multimodal datasets as well as 3 new datasets created for MathVista, where each dataset contains different styles of questions. There are both multiple-choice questions and free-form questions, where the free-form questions ask for a response in the form of an integer, floating-point number, or list.

<!-- Contributors: Automatically Generated -->
Contributed by [@ShivMunagala](https://github.com/ShivMunagala)
<!-- /Contributors: Automatically Generated -->

<!-- Usage: Automatically Generated -->
## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Or, if developing on a clone of the `inspect_evals` repo, you can install the package in editable mode with:

```bash
pip install -e ".[dev]"
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/mathvista --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```
<!-- /Usage: Automatically Generated -->

<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/mathvista --limit 10
inspect eval inspect_evals/mathvista --max-connections 10
inspect eval inspect_evals/mathvista --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->

## Dataset
Here is an example from the dataset:

>**Question:** The derivative of f(x) at x=2 is ____ that at x=5
>
>**Choices:** "larger than", "equal to", "smaller than"
>
>**Image:**
>
>![Image for example question](example.png)
>
>**Correct answer:** "equal to"

## Scoring
An accuracy is calculated over the datapoints.
