# LUNGUAGE: A BENCHMARK FOR STRUCTURED AND SEQUENTIAL CHEST X-RAY INTERPRETATION

## Installation for Structuring Framework of Chest X-ray Report

First, install the required environment:

```bash
pip install -r requirements.txt
```

<details>
<summary><span style="font-size:2em; font-weight:700"> Preliminaries: Extracting Report Sections (MIMIC-CXR)</span></summary>


Use `dataset/create_section_file.py` to parse MIMIC-CXR radiology reports into sections and export them to CSV.

If your environment stores MIMIC-CXR under one of the known defaults, simply run:

```bash
cd dataset
python create_section_file.py
```

The script will try these report directories in order and use the first that exists:
- `/nfs_data_storage/mimic-cxr-jpg/2.0.0/reports`
- `/nfs_data_storage/mimic-cxr-dcm/reports`
- `/home/data_storage/mimic-cxr-jpg/2.0.0/reports`

It will also, by default, look for the split and metadata files at:
- Split: `/nfs_data_storage/.../mimic-cxr-2.0.0-split.csv.gz` then `/home/data_storage/.../mimic-cxr-2.0.0-split.csv`
- Metadata: `/nfs_data_storage/.../mimic-cxr-2.0.0-metadata.csv.gz` then `/home/data_storage/.../mimic-cxr-2.0.0-metadata.csv`

### Customizing paths

You can override any of the paths via CLI options:

```bash
python dataset/create_section_file.py \
  --reports_path /path/to/mimic-cxr/reports \
  --split_path /path/to/mimic-cxr-2.0.0-split.csv.gz \
  --metadata_path /path/to/mimic-cxr-2.0.0-metadata.csv.gz \
  --output_path ./processed_reports \
  --no_split
```

### Arguments

- `--reports_path`: Path to the MIMIC-CXR reports directory. If omitted, the script tries known defaults (see above).
- `--output_path`: Directory to write outputs. Default: `./processed_reports`.
- `--no_split`: If provided, writes a single `mimic_cxr_sections.csv` instead of chunked files.
- `--split_path`: Path to the split CSV (supports `.csv` or `.csv.gz`). Default: tries known locations.
- `--metadata_path`: Path to the metadata CSV (supports `.csv` or `.csv.gz`). Default: tries known locations.

### Outputs

- `processed_reports/mimic_cxr_sectioned.csv`: Wide CSV of full report text and per-section fields.
- `processed_reports/mimic_cxr_reports.csv`: Long-form CSV with columns `subject_id, study_id, sequence, split, StudyDate, StudyTime, section, report` where `section` ∈ {`hist`, `find`, `impr`}.
- If `--no_split` is not used, sharded files `mimic_cxr_XX.csv` containing study-level section text are written.
</details>


<details>
<summary><span style="font-size:3em; font-weight:700"> Dataset Preparation</span></summary>


First, check whether the dataset folder already exists. If it does not, please download the following two files from PhysioNet:

- `Lunguage.csv`
- `Lunguage_vocab.csv`

```
project_root/
├── dataset/
│ ├── Lunguage.csv
│ └── Lunguage_vocab.csv
```
</details>

<details>
<summary><span style="font-size:3em; font-weight:700"> Structuring Framework with MedGemma & GPT-OSS with vLLM</span></summary>

1. Install and run GPT-OSS or MedGemma with vLLM [reference](https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html#gpt-oss-vllm-usage-guide)

2. Run framework
```
python singleSR/run.py --deployment_name [docker_name] --api_key none --mode gold_eval --n_retrieval 5 --port [port_number]
python sequentialSR/llm_run.py --LLM_name [docker_name] --api_key none --model_run --process_missing --port [port_number] --few_shot

```
</details>


<details>
<summary><span style="font-size:3em; font-weight:700"> API Run</span></summary>

The framework currently supports OpenAI GPT and Fireworks AI for Deepseek-v3, Qwen3, Llama4-maverick. You'll need API keys to run the structuring steps.

## Quick Run

### 1. Single Report Structuring
To run the first step of the framework for single report structuring:

```bash
./1.single_runner.sh
```

### 2. Sequential Report Structuring
To run the second step of the framework for sequential report structuring:

```bash
./2.sequential_runner.sh
```

### 3. Metric Evaluation
To run the third step of the framework for metric evaluation:

```bash
./3.metric_runner.sh
```
</details>



## LunguageScore Metric

Use the standalone script `lunguagescore.py` to compute only LunguageScore for a model’s structured outputs.

### Single-report mode
```bash
python lunguagescore.py --mode single --model_name [target_model_name]
```

### Sequential mode
```bash
python lunguagescore.py --mode sequential --model_name [target_model_name]
```

### ReXVal mode
```bash
python lunguagescore.py --mode rexval --rexval_path benchmark/rexval.csv
```

### Options
- `--model_name`: one of `maira_standard, maira_cascade, rgrg, cvt2distilgpt2, medversa, medgemma, lingshu`
- `--gt_path`: path to gold structured CSV (default: `dataset/gold_SR.csv`)
- `--benchmark_sr_dir`: directory containing `<model_name>_SR.csv` (default: `benchmark_SR`)
- `--rexval_path`: path to structured ReXVal CSV (default: `benchmark/rexval.csv`)
- `--output`: custom output CSV path (default: saves under `scores/`)
- `--print_head`: print the head of the result instead of only the save path
