# SWE-bench

This is an inspect-native implementation of [the SWE-bench dataset](https://www.swebench.com/), a benchmark for testing the ability of AI agents to solve software engineering tasks. 

<!-- Contributors: Automatically Generated -->
Contributed by [@max-kaufmann](https://github.com/max-kaufmann)
<!-- /Contributors: Automatically Generated -->

<!-- Usage: Automatically Generated -->
## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install "inspect_evals[swe_bench]@git+https://github.com/UKGovernmentBEIS/inspect_evals"
```

Or, if developing on a clone of the `inspect_evals` repo, you can install the package in editable mode with:

```bash
pip install -e ".[dev,swe_bench]"
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/swe_bench --model openai/gpt-4o
inspect eval inspect_evals/swe_bench_verified_mini --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```
<!-- /Usage: Automatically Generated -->

>[!NOTE]
>When first running the swe_bench task, it will build the necessary docker images. This can be resource intensive - for the full swe_bench Verified split, up to several hours, and ~280GB of storage.
>
>SWE-bench will take a while to run, and uses a lot of tokens. If things are too slow, you should increase the level of parallelism - see https://inspect.ai-safety-institute.org.uk/parallelism.html. Note that running too many docker containers on your machine can also cause issues, most notably with a 'ALL PREDEFINED ADDRESS POOLS HAVE BEEN FULLY SUBNETTED' error - we don't recommend running more than 32 containers at any one time.


<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/swe_bench --limit 10
inspect eval inspect_evals/swe_bench_verified_mini --max-connections 10
inspect eval inspect_evals/swe_bench --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->

## Dataset


You will be provided with a partial code base and an issue statement explaining a problem to resolve.

> **Issue**: napoleon_use_param should also affect "other parameters" section
> 
> Problem:
> 
> Currently, napoleon always renders the Other parameters section as if napoleon_use_param was False, see source
>     ```
>     def _parse_other_parameters_section(self, section):
>         # type: (unicode) -> List[unicode]
>         return self._format_fields(_('Other Parameters'), self._consume_fields())
> 
>     def _parse_parameters_section(self, section):
>         # type: (unicode) -> List[unicode]
>         fields = self._consume_fields()
>         if self._config.napoleon_use_param:
>             return self._format_docutils_params(fields)
>         else:
>             return self._format_fields(_('Parameters'), fields)
>     ```

## Other Notes

### Running the benchmark
``swe_bench.py``` file contains ```swe_bench``` function, which creates an instance of a SWE-bench Task:

```python
from inspect_ai import eval
from inspect_ai.solver import system_message, basic_agent
from inspect_ai.tool import bash, python
from inspect_evals.swe_bench import swe_bench

# Create an agent that only uses bash, and an agent that only uses python
agent = basic_agent(tools=[bash(timeout=30)])
task = swe_bench(
    dataset="princeton-nlp/SWE-bench_Verified",
    solver=agent,
    instance_ids=["sympy__sympy-22914", "sympy__sympy-13878"],
)
# Compare how these two agents perform.
eval(task, model="openai/gpt-4o")
```

### Comparing to official swe-bench baselines
Submissions to [the official SWE-bench leaderboard](https://www.swebench.com/) often come with a set of trajectories and per-swebench-instance results. To download these baselines, you can git clone the official baselines repository, and copy the baselines to a local directory:
```bash
# Download the swe-bench baselines. This will take a while. 
if [ ! -d '/tmp/swebench_baselines_repo' ]; then
    git clone https://github.com/swe-bench/experiments /tmp/swebench_baselines_repo
fi
# Copy to current directory
mkdir -p swebench_baselines
cp -r /tmp/swebench_baselines_repo/evaluation/verified/20240620_sweagent_claude3.5sonnet ./swebench_baselines/
# Can add other files in /tmp/swebenchbaselines/evaluations/... here
```

This means that you can compare the performance of your agent not just on the full dataset, but also on subsets of that dataset. This is often useful when trying to run smaller, cheaper experiments. To make this comparison easy, we provide a scorer that will calculate the average performance of an existing baseline on that set of trajectories:

```python
from inspect_evals.swe_bench import swebench_baseline_scorer
agent = basic_agent(tools=[bash(timeout=30)])
scorers = [
    swebench_baseline_scorer(
        "./swebench_baselines/20240620_sweagent_claude3.5sonnet",
        name="SWE-agent_3.5_sonnet",
    ),
    swebench_scorer(),
]
task = swe_bench(
    dataset="princeton-nlp/SWE-bench_Verified",
    solver=agent,
    instance_ids=["sympy__sympy-22914", "sympy__sympy-13878"],
    scorer=scorers,
)

eval(task, model="openai/gpt-4o")
```

This will lead to both numbers being reported in the final output, allowing you to compare baselines:

![SWE-bench baseline comparison](./docs/swebench_comparison.jpeg)

### Parity with the original implementation

We keep track of any known issues with our scoring [here](https://github.com/UKGovernmentBEIS/inspect_evals/issues/36). We recommend that before submitting to the leaderboard, or comparing to public results in a paper, you use the ```save_outputs_to_swebench_format``` function to score with the original implementation:

```python
from inspect_evals.swe_bench import save_outputs_to_swebench_format
logs = eval(task = swe_bench,solver=agent)
save_outputs_to_swebench_format(logs, "./swebench_formatted_outputs/")
```

You can then use the CLI to score these with the original implementation (as described [in their README](https://github.com/princeton-nlp/SWE-bench)):

```bash
python -m swebench.harness.run_evaluation \
    --predictions_path path-to-outputs \
    --max_workers 4 \
    --run_id check-outputs
```
