# Table Benchmark

## Data Path

- Task file is in the `symDataset/tasks/TableQA/dataset.sqlite`.

- The scaled dataset is in the `symDataset/scaledDB/` path.

## Data Format

The scaled dataset is separated into 4 scales `8k` to `64k` , all of which are sampled from `128k` . Every scale has 10 different databases and every database have 10 different sample instances (indexing from 0 to 9) .

The task database have ten tables, each table has the following schema

```SQL
CREATE TABLE {database_name} (
    scale TEXT,           /* 8k to 64k */
    dbIdx INTEGER,        /* 0 to 9 */
    sampleIdx INTEGER,    /* 0 to 9 */
    questionIdx INTEGER,  /* 0 to 13 */
    qtype TEXT,           /* 7 types detailed in the paper */
    question TEXT,        /* question instances generated from the template */
    rightIdx INTEGER,     /* 0 to 3 refers right choice is A to D */
    A TEXT,               /* Choice A */
    B TEXT,               /* Choice B */
    C TEXT,               /* Choice C */
    D TEXT,               /* Choice D */
    PRIMARY KEY (scale, dbIdx, sampleIdx, questionIdx)
);
```

The `{database_name}` refers to the 10 databases mentioned in the scale dataset above. The `scale, dbIdx, sampleIdx, questionIdx` can be considered 4 dimensions of a question. The template question is in the `symbolic/` folder.

## Run

- The core sample logic is in `benchmarkUtils/database.py`.

- The symbolic extension code is in the `symbolic/` path.

- The single table evaluation is in `unionTableBench.py`.

- The core evaluation code for Experiment 2-4 is in the `symDataloader/utils.py` . You can refer the example `symDataloader/gpt4o.py`.

- The SQL evaluation code can be found in the `tqabench/` path.
