# AI2 Reasoning Challenge (ARC)

[ARC](https://arxiv.org/pdf/1803.05457) is a benchmark using natural science questions to evaluate a model's knowledge and reasoning capabilities. The dataset ships with `Easy` and `Challenge` sets.

<!-- Contributors: Automatically Generated -->
Contributed by [@jjallaire](https://github.com/jjallaire)
<!-- /Contributors: Automatically Generated -->

<!-- Usage: Automatically Generated -->
## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install "inspect_evals[math]@git+https://github.com/UKGovernmentBEIS/inspect_evals"
```

Or, if developing on a clone of the `inspect_evals` repo, you can install the package in editable mode with:

```bash
pip install -e ".[dev,math]"
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/arc_easy --model openai/gpt-4o
inspect eval inspect_evals/arc_challenge --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```
<!-- /Usage: Automatically Generated -->

<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/arc_easy --limit 10
inspect eval inspect_evals/arc_challenge --max-connections 10
inspect eval inspect_evals/arc_easy --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->

## Dataset

The ARC dataset is a dataset of 7,787 genuine grade-school level, multiple-choice science questions. Here is an example prompt (after being further process by Inspect):

> Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: $LETTER (without quotes) where LETTER is one of A,B,C,D.
>
> An astronomer observes that a planet rotates faster after a meteorite impact. Which is the most likely effect of this increase in rotation?
>
> A) Planetary density will decrease.  
> B) Planetary years will become longer.  
> C) Planetary days will become shorter.  
> D) Planetary gravity will become stronger.  

The model is then tasked to pick the correct choice.

## Scoring
A simple accuracy is calculated over the datapoints.
