# DCA-Benchmark


DCA-Benchmark aims to provide a comprehensive benchmark for evaluating LLM agents' capabilities in discovering data quality issues across online dataset platforms, representing the first step of the curation pipeline. Throughout this document, we will refer to such an LLM agent as a **"Curator"** to highlight its role in this task. A well-performing Curator can detect and locate existing issues, which is critical for subsequent fixes by human maintainers or other LLM agents. We collected 221 representative samples from 8 online dataset platforms and classified them into 4 types with 18 tags according to their various content and difficulty.


## Key Features

- **Real-world Cases with Minimal Simplification**: All test cases in DCA-Benchmark have references to real-world sources, allowing benchmark users to understand them better in practical scenarios. To test the Curator's ability in complex real-world environments, DCA-Benchmark includes all relevant dataset files for each test case, not just the flawed ones.

- **Multiple Difficulty Levels**: DCA-Benchmark provides four levels of hints for each test case in the benchmark. With higher-level hints, the Curator gains more information about the content and location of the issue. This approach aims to make the task more achievable and gauge the information required for the Curator to detect these issues.

- **Accurate Automatic Evaluation**: Unlike traditional machine learning tasks, dataset curation does not have labels that can be directly evaluated by scripts. Human-level efforts are required to rate the Curator's performance, which is not scalable. Therefore, we developed an automatic and accurate evaluation scheme using GPT-4 to replace human annotators.

## Getting Started

**Install the required packages using:**

```bash
conda create -n dca-bench python=3.9
conda activate dca-bench
pip install -r requirements.txt
```

You can see meta information of each test cases in `benchmark/task_meta.json`, where each test case has three important attributes:

- **`id`**: A globally unique identifier for each test case, used to find the corresponding file locations.
- **`scenario`**: Indicates the platform from which the case is collected.
- **`hints`**: Each test case has four different level hints, providing ascending levels of information.

**Set up Your OpenAI Key**
Please refer to https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety to set up your OpenAI key.

## Benchmark API

The `benchmark API` is used to format the input for the Curator and obtain comprehensive statistics for DCA-Benchmark. You can see `api_demo.ipynb` for some use cases.

### Getting Test Case Input for the Curators

You can get the input of a test case for your Curator using the following code:

```python
from benchmark.api import BenchmarkManager

manager = BenchmarkManager()
hints, dataset_files, doc_paths = manager.get_test_case_info(id, dest_dir, use_doc=False)
```

Here are the return types for the function `get_test_case_info`:

- **`hints`**: `List[str]` – The list of hints for the test case.
- **`dataset_files`**: `List[str]` – The list of paths to the dataset files for the test case.
- **`doc_paths`**: `List[str]` – The list of paths to the document materials for the test case.

Params:

- **`dest_dir`**: If not `None`, `dataset_files` for this case will be copied to `dest_dir`.
- **`use_doc`**: If set to `True`, returns the paths of document materials for this scenario. This option was not used in the paper.

### Curator Output Format

Although the design of the Curators is not restricted, we have provided an example of output format to better align with the Evaluator, which requires three fields:

```json
{
  "issue": "Wrong name in README",
  "evidence": "# My cool task This is a description of my cool task... in README",
  "description": "The name of the task in the README is incorrect. It should be 'My awesome task' instead of 'My cool task'."
}
```

You can use `manager.format_output(issue, evidence, description)` to get the formatted output. Or use `manager.get_output_format()` to get the example above as a one-shot example for your Curator. **It's also ok to render them in an item list**, the point here is to include three fields, "issue" for a brief summary, "evidence" for contextual location in the file where the issue occurs, and "description" for a detailed analysis, just like a Github Issue.

## Evaluation

The evaluation results used in Tab.4 and Fig.4 of the paper are under `pipeline/eval_log`, where `eval=` indicates which model is used as Evaluator, `test=` denotes the Curator.

You could run the following code to get the results shown in Fig. 4 of the paper, which evaluates the performance of the baseline Curator based on the OpenAI Assistant API:

```bash
python -m pipeline.run_evaluation
```

The default results will be shown under `pipeline/eval_log/eval=gpt-4-0125-preview/test=gpt-4-0125-preview/example_run`.

**To evaluate your own Curator's performance**, structure your results as follows:

```
pipeline/output/{your_model_name}/{scenario}/{id}/hint_level_{hint_level}/output.txt
```

Then run:

```bash
python -m pipeline.run_evaluation --test_model_id=your_model_name
```

To test the performance of your Curator on a single (case, hint) pair, use the following code:

```python
from benchmark.api import Evaluator
evaluator = Evaluator()

result, cost = evaluator.run_single_evaluation(id, hint_level, input_path, save_dir)
```

- `id`: The file ID of the test case.
- `hint_level`: The hint level, which can be 0, 1, 2, or 3.
- `input_path`: The path to the output of your Curator for this (case, hint) pair.
- `save_dir`: The directory where you want to save the evaluation results.

## Try Baseline Curator

The test outputs from several baseline Curators are under `pipeline/output`. To try the Baseline Curator (OpenAI Assistant API) yourself, run:

```bash
python -m pipeline.run_inference
```

The default results will be shown at `pipeline/example_output/gpt-4-0125-preview`

## Contribute new datapoint

To contribute a new datapoint to DCA-Bench, you could run `contribute.py` to add a new datapoint, or you could simply write in `benchmark/task_meta.json`.

## Others

- `tools/kaggle_scrapper` and `tools/huggingface_scrapper` are used to collect the dataset from Kaggle and Huggingface respectively.


# Reproduction Guidelines

After setting up the environment, please download `platforms` from Google Drive: https://drive.google.com/file/d/1QbyH5j3obLtpa9vxO-NINg0_8iv1w9tQ/view?usp=sharing

Put it under `benchmark/` folder and unzip it. The overall structure should be like:

```
benchmark/
├── platforms/
│   ├── kaggle/
│   ├── huggingface/
│   ├── ...
```

## Experiment Notebooks
- Statistics of DCA-Bench (Section 3.1, Tab. 1): `notebooks/statistics.ipynb`
- Human aligment study (Section 4.1, Tab. 2): `notebooks/human_alignment.ipynb`
- Evaluation of baseline curator (Section 4.2, Fig. 3): `notebooks/eval_of_baseline_curator.ipynb`
- Consistent Evaluation (Section 4.1): `notebooks/std_of_evaluator.ipynb`
- Webpage experiment (Section 4.2, Tab. 3): `notebooks/webpage_expr.ipynb`
- Bias study (Section A.2.4):
  - Self-preference bias study: `notebooks/self_preference_bias.ipynb`
  - Length bias study: `notebooks/length_bias.ipynb`
- Cost analysis script and results (Section A.5.3): `notebooks/cost_calculator.ipynb`
- Content length versus success rate (Section A.3.1, Fig. 4): `notebooks/content_length.ipynb`

## Artifacts
- `pipeline/gpt_3.5_turbo_labels_run.json`: human labeled results from GPT-3.5-turbo based Curator on 92 (case, hint) pairs.
- `pipeline/label.json`: human labeled results from GPT-4-0125-preview based Curator on 92 (case, hint) pairs.
- `pipeline/webpage.json`: hard set for webpage ChatGPT experiment.
- `dca-bench.csv`: the source and license information of test cases in DCA-Bench.

# License

DCA-Benchmark will be released under the Apache License 2.0. The dataset files in `platforms` are collected from Kaggle, Huggingface, and other platforms, and their licenses have been carefully handled. For more details on licensing, please refer to the `dca-bench.csv` file.