# MBPP: Mostly Basic Python Problems

[MBPP](https://arxiv.org/abs/2108.07732) is a dataset for evaluating the ability of models to synthesize short Python programs from natural language descriptions. It contains programming tasks that are designed to be solvable by entry-level programmers. We evaluate on the [sanitized test split](https://huggingface.co/datasets/google-research-datasets/mbpp/viewer/sanitized/test) of the dataset, which was hand-verified to remove samples that lacked detail or were ambiguous.

<!-- Contributors: Automatically Generated -->
Contributed by [@jddantes](https://github.com/jddantes)
<!-- /Contributors: Automatically Generated -->

<!-- Usage: Automatically Generated -->
## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Or, if developing on a clone of the `inspect_evals` repo, you can install the package in editable mode with:

```bash
pip install -e ".[dev]"
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/mbpp --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```
<!-- /Usage: Automatically Generated -->

<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/mbpp --limit 10
inspect eval inspect_evals/mbpp --max-connections 10
inspect eval inspect_evals/mbpp --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->

## Dataset
Here is a sample from the dataset:

> Write a python function to find the volume of a triangular prism.
>
> test_list:
> ```
> [ "assert find_Volume(10,8,6) == 240",
>    "assert find_Volume(3,2,2) == 6",
>    "assert find_Volume(1,2,1) == 1" ]
> ```

The model is tasked to write Python code that will pass the assert statements in the `test_list`.

## Scoring
The model is prompted to solve the problem, given its description and test cases to pass. Additionally, [few shot examples](https://github.com/google-research/google-research/tree/master/mbpp) can be included in the prompt, with prompts patterned after an [AgentCoder implementation](https://github.com/huangd1999/AgentCoder/blob/main/prompts/mbpp_prompt.txt) which topped the [leaderboard](https://paperswithcode.com/sota/code-generation-on-mbpp).

Evaluation metrics follow similar benchmarks such as HumanEval. The benchmark uses the $\text{pass}@k$ metric to measure functional correctness. In brief terms, this is the per-problem probability of at least 1 correct sample generation given $k$ generations. It is defined using the following expectation:

$$ \text{pass@}k := \underset{\text{Problems}}{\mathbb{E}}\left[1-\frac{{n-c}\choose{k}}{{n}\choose{k}}\right]$$

where we sample $n \geq k$ generations to reduce variance. Note that the default in this benchmark implementation is $n = 5$, and we evaluate $\text{pass}@k$ for $k \in \\{1, 2, 5\\}$.
