# BigCodeBench

BigCodeBench is a coding benchmark with a special focus on models' use of Python libraries.

- [Arxiv paper](https://arxiv.org/abs/2406.15877)
- [GitHub repo](https://github.com/bigcode-project/bigcodebench)
- [Website](https://bigcode-bench.github.io/)
- [Official leaderboard](https://huggingface.co/spaces/bigcode/bigcodebench-leaderboard)

Currently, this eval pulls from version v0.1.2 of the dataset from [huggingface](https://huggingface.co/datasets/bigcode/bigcodebench), which has a total of 1140 questions. As of Nov 25, 2024, gpt-4o-mini gets 55.4% of them correct in this inspect implementation[^1] (compared to 57.4% on the official leaderboard). Running gpt-4o-mini took a total of 372,319 input tokens and 799,339 output tokens.

This implementation copies heavily from the humaneval implementation in inspect. The biggest difference is that it requires access to several python packages, which are installed through the Dockerfile. The Dockerfile in this folder deviates from the [official docker files](https://github.com/bigcode-project/bigcodebench/blob/main/Docker/Evaluate.Dockerfile) by installing tensorflow without specifying the version, to make it platform-independent.

Instructions for BigCodeBench come in two versions, complete_prompt and instruct_prompt. The complete prompt contains package imports and a Python function header with some examples.

Example *complete* prompt, taken from BigCodeBench/1116:

```python
import random
import statistics

# Constants
AGE_RANGE = (22, 60)

def task_func(dict1):
 """
 Calculate the mean, the median, and the mode(s) of the age of the employees in the department "EMP$$." 
 Generate random ages for each employee within the range [22, 60].

 Parameters:
 dict1 (dict): A dictionary with department codes as keys and number of employees 
 as values.

 Returns:
 tuple: A tuple of mean, median, and a list of mode(s) of employee ages.

 Requirements:
 - random
 - statistics

 Example:
 >>> random.seed(0)
 >>> d = {'EMP$$': 10, 'MAN$$': 5, 'DEV$$': 8, 'HR$$': 7}
 >>> stats = task_func(d)
 >>> print(stats)
 (44.7, 46.5, [46, 48, 24, 38, 54, 53, 47, 41, 52, 44])
```

And the *instruct* version:

> Calculate the mean, the median, and the mode(s) of the age of the employees in the department "EMP$$." Generate random ages for each employee within the range [22, 60].
>
> The function should output with:
>     tuple: A tuple of mean, median, and a list of mode(s) of employee ages.
> You should write self-contained code starting with:

```
import random
import statistics
# Constants
AGE_RANGE = (22, 60)
def task_func(dict1):
```

Since the complete prompt contains examples not seen in the instruct prompt, this implementation always uses the complete prompt, prefaced with the following (customizable) instruction prompt:

> Read the following function signature and docstring, and fully implement the function described. Make sure to include the import statements and use the same variable names as stated in the header.

By default, this evaluation checks the pass@1 rate (i.e., the model has one chance of getting the question correct).

<!-- Contributors: Automatically Generated -->
Contributed by [@tim-hua-01](https://github.com/tim-hua-01)
<!-- /Contributors: Automatically Generated -->

## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/bigcodebench --model openai/gpt-4o
```

This would automatically setup docker containers, which takes around 12 minutes. We recommend first running docker to setup the environment (which would then be cached) before running the evaluation:

```bash
cd src/inspect_evals/bigcodebench
docker compose build
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```\

<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/bigcodebench --limit 10
inspect eval inspect_evals/bigcodebench --max-connections 10
inspect eval inspect_evals/bigcodebench --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->

## Scoring

We run the model submitted answer in a docker sandbox and check whether it passes unit tests in the BigCodeBench dataset.

## Errata

As with all evaluations datasets, there are mistakes in the data (see e.g., the issues with the HumanEval dataset [here](https://openreview.net/forum?id=1qvx610Cu7)). As of Nov 26, 2024 (v. 0.1.2.), we've found the following potential bugs:

- Task 111: The provided answer does not pass all test cases when run locally.
- Task 227: [The libsndfile c library](https://github.com/libsndfile/libsndfile) seems to be required for this task to work? It's currently running into an import issue.
- Task 917, it looks plausible that the code would have floating point errors that get flagged as actual errors. Running gpt-4o's code results in:

```
FAIL: test_case_1 (main.TestCases)
Traceback (most recent call last):
File "<string>", line 85, in test_case_1
AssertionError: 110.99401141660307 != 110.99999292499156 within 2 places (0.0059815083884871 difference)

======================================================================
FAIL: test_case_2 (main.TestCases)
Traceback (most recent call last):
File "<string>", line 106, in test_case_2
AssertionError: 211.99463480210375 != 211.99999982088116 within 2 places (0.005365018777411024 difference)

======================================================================
FAIL: test_case_4 (main.TestCases)
Traceback (most recent call last):
File "<string>", line 147, in test_case_4
AssertionError: 412.9939192324403 != 412.9999964967954 within 2 places (0.006077264355099032 difference)
```

[^1]: A slightly different instruction prompt was used for this evaluation.
