# Benchmarking Intelligent LLM Agents for Conversational Data Analysis


[![License](https://img.shields.io/badge/License-CC%20By%20SA%204.0-orange.svg)](https://creativecommons.org/licenses/by-sa/4.0/deed.en)
[![Python 3.10+](https://img.shields.io/badge/Python-3.10+-teal.svg)](https://www.python.org/downloads/release/python-390/)
[![OpenAI 0.28.1](https://img.shields.io/badge/OpenAI-0.28.1-beige.svg)](https://www.python.org/downloads/release/python-390/)


<p align="center" width="100%">
<a><img src="material/decision_company.png" style="width: 100%; min-width: 300px; display: block; margin: auto;"></a>
</p>

## Project Overview

Tapilot-Crossing is an innovative benchmark designed to evaluate Language Model (LLM) agent performance on conversational data analysis tasks. This project introduces a cost-effective way to simulate realistic user-agent conversation via DECISION COMPANY. Tapilot-Crossing includes 1024 human-machine conversation with 1176 user intents across four scenarios: **Normal**, **Action**, **Private**, and **Private Action**. It also features the **A**daptive **C**onversation **R**eflection (**ACR**) strategy, aimed at improving LLM agent ability to learn from their conversation histories, leading to significant performance enhancements. **Please NOTE that any word "interaction" appears in this project means "conversation" or "dialogue".**

## Dataset Structure

This section outlines the structure of the `data` directory, detailing its organization across several layers and describing the contents of each.

### 1. Directory structure of `interaction_data`

### First Layer: Dataset Domains
The top level of the dataset directory is organized by dataset domains, each representing a specific area of interest. For example, `atp_tennis` is dedicated to tennis matches.

### Second Layer: Agent Profiles and Conversation Length
The next level categorizes data by agent profiles and the length of conversation:
- **Long Conversation**: Represented by profiles with extensive experience, such as "Carlos" who is noted for his expertise in organizing tennis tournaments.
- **Short Conversation**: Denoted by a prefix `short_`, indicating brief conversation with the agent, e.g., `short_Carlos`.

### Third Layer: Data Modes
Data is further organized into four modes reflecting the nature of the conversation turns:
- **Action**: Conversation turns featuring special actions (e.g., analysis, clarification requests).
- **Action_Private**: A private library version of the Action mode.
- **Normal**: Conversation turns with clear user intents that do not require any special actions.
- **Private**: A private library version of the Normal mode.

### Fourth Layer: Conversation Turns
The final layer is organized by the conversation turns themselves, providing a granular view of the dataset.

### 2. Content Files of `data`
- `interaction_data_json`: Contains all the conversation data of tapilot data in **JSONL** format. They are divided into four modes: 1. Normal Mode (`normal.jsonl`); 2. Private Mode (`private.jsonl`); 3. Action Mode (5 different action types, e.g., `action_plotqa.jsonl`); 4. Private Action Mode (`private_action_correction.jsonl`).
- `interaction_data`: Contains all the conversation data of tapilot data. For each conversation turn (e.g., `turn_1_short_1` for short conversation, `turn_1` for long conversation), the following files are included:
  - `meta_data.json`: Contains metadata about the problem, including `result_type`, `action`, and `data_id`.
  - `ref_result` and `pred_result` directories: Store files (pickles or PNGs) generated from the reference and prediction codes, respectively.
  - `prompt_curr.txt`: The basic prompt for the current conversation turn used for querying large language models.
  - `prompt_curr_oracle.txt` (Optional): An enhanced version of the basic prompt that includes oracle private functions for the current conversation turn.
  - `ref_code.txt`: The ground truth solution code for the current conversation turn.
  - `eval.py`: The evaluation code specific to the current conversation turn.
  - `reference_answer.txt` (Optional): The correct answer for the current conversation turn in multi-choice format.
  - `reference` directory: Contains the conversation history and reference code:
    - `prompt_code_hist.json`: Basic prompt with conversation history in a list of dictionaries format, adhering to the standard GPT input format.
    - `prompt_code_hist.txt`: Plain text format of the basic prompt with conversation history.
    - `ref_code_hist.py`: Reference code of conversation history excluding the current turn's reference code, useful as code context for model-generated code.
    - `ref_code_all.py`: Includes the complete reference code history along with the current turn's code, serving as a comprehensive reference that can generate results for the current turn.
- `resource`: Contains all the tabular data in csv file of 5 domains of tapilot data; along with the private library named "decision_company" in python and json format.
- `rdb`: Contains the History Relational Database. We split the User-AI conversation into several singleturn user queries and AI answers stored in a relational database, indexed by the conversational order. This storage is subject to dynamic combinations for different scenarios. 

The visualization of the `tapilot_data` directory structure which should be stored in **`data/`** folder of this project.

```
data
├── interaction_data_json
  ├── action_analysis.jsonl
  ├── action_bg.jsonl
  ├── action_correction.jsonl
  ├── action_plotqa.jsonl
  ├── action_una.jsonl
  ├── normal.jsonl
  ├── private_action_correction.jsonl
  └── private.jsonl
├── interaction_data
  ├── atp_tennis
      ├── Carlos
      ├── Emma
      ├── John
      ├── short_Carlos
      ├── short_Emma
      └── short_John
          ├── short_action
          ├── short_action_private
          ├── short_normal
          └── short_private
              ├── turn_1_short_1
              ├── turn_2_short_1
              ├── turn_2_short_2
                  ├── meta_data.json
                  ├── eval.py
                  ├── ref_code.py
                  ├── prompt_curr_oracle.txt (Optional: only for private mode)
                  ├── prompt_curr.txt
                  ├── ref_result
                  ├── pred_result
                  ├── reference_answer.txt (Optional: only for multi-choice mode)
                  └── reference
                      ├── prompt_code_hist.json
                      ├── prompt_code_hist.txt
                      ├── ref_code_hist.py
                      └── ref_code_all.py (Optional: only for code generation mode)
  ├── credit_card_risk
  ├── fast_food
  ├── laptop_price
  └── melb_housing
├── resource
└── rdb
```

## Environment Setup

Before starting, ensure Python 3.10 is installed on your system. Some Python libraries are required which can be installed via pip:

```bash
conda create -n tapilot python=3.10
source activate tapilot
pip install -r requirements.txt
```

## Collect Agent Response 
### Generate responses
To evaluate the effectiveness of different Large Language Model (LLM) agents in various settings, execute them with the designated prompts and document their responses. The following steps can guide you through this process:

  - For code gen mode evaluation:

    ```bash
    sh ./run/run_code_gen.sh
    ```

  - For multi-choice mode evaluation:

    ```bash
    sh ./run/run_multi_choice.sh
    ```

This step usually generates output in a json file located in [`./output/`](./output/), where each key is the data path and value is LLM response. We include examples in the output.

### Postprocessing for LLM responses

To extract Python code from LLM responses and integrate it into the corresponding code history, preparing it for execution. Different models with different prompt may vary at this part.
  - For baseline model:

    ```bash
    sh ./run/postprocess_code_gen.py
    ```

This process creates a file named `pred_code.py` containing extracted Python code. This file is saved within the folder corresponding to each dialogue turn, facilitating easier evaluation.

## Performance Evaluation
   
### Generate Results for Evaluation
First, you need to generate results from predicted and reference code. 

  - For referenced results generation:
    
    ```bash
    sh ./run/run_reference_code.sh
    ```

  - For prediction results generation:
    
    ```bash
    sh ./run/run_prediction_code.sh
    ```

This step will execute the reference or prediction code, producing results that are saved at **`ref_result/`** or **`pred_result/`** under each dialogue turn's folder for evaluation.

### Evaluation on Each Setting
To quantitatively evaluate the performance of different models based on the results obtained from the prediction code execution. Utilizing specific `eval.py` within each dialogue turn's folder, this step facilitates a comparative evaluation of the results.

  - For code generation setting:

    ```bash
    sh ./run/eval_code_gen.sh
    ```

  - For analysis setting:

    ```bash
    sh ./run/eval_multi_choice.sh
    ```

This step generates `eval_stats.json`, which contains detailed performance evaluation located under each dialouge turn folder.

### Performance Report
To compile and summarize the performance metrics across all settings, providing a comprehensive overview of the LLM agents' capabilities. You can use:

  ```bash
  sh ./run/eval_overall.sh
  ```

## Code License

This project code is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
