# PAWS

[PAWS](https://arxiv.org/pdf/1904.01130) is a dataset designed for evaluating models on the task of paraphrase detection by providing pairs of sentences that are either paraphrases or not, facilitating the training and benchmarking of natural language understanding systems.

<!-- Usage: Automatically Generated -->
## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Or, if developing on a clone of the `inspect_evals` repo, you can install the package in editable mode with:

```bash
pip install -e ".[dev]"
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/paws --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```
<!-- /Usage: Automatically Generated -->


<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/paws --limit 10
inspect eval inspect_evals/paws --max-connections 10
inspect eval inspect_evals/paws --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->

## Dataset

Here is an example from the dataset (after processing by Inspect):

> Answer Yes if the following two sentences are paraphrases. If they are not, answer No. 
> Do not give any other answer other than Yes or No in your response. 
> Sentence1: These geographically correspond to the traditional Glasgow District , South , Edinburgh District , and North and Midlands districts .
> Sentence2: These geographically roughly correspond to the traditional districts of Glasgow District , South , Edinburgh District and North and Midlands .
> Yes

## Scoring
A simple accuracy is calculated over the datapoints.
