# T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generation




<a name="prompt_suite"></a>
## T2I-ReasonBench Prompt Suite
The T2I-ReasonBench prompt suite includes 800 prompts covering 4 dimensions, each with 200 prompts. 

For each dimension, the text prompts used to generate the images are saved in a json file under the ```prompts/``` directory.

The prompt-specific question-criterion pairs used to assist the evaluation are saved in the corresponding json file under the ```deepseek_evaluatioin_qs/``` directory.

The pipeline prompts are saved in the corresponding json file under the ```pipeline_prompts/``` directory.


<a name="installation"></a>
## Installation

We adopt Qwen2.5-VL as our evaluation tool.
Please set the environment variable manually as follows:
```
conda create -n qwen python==3.11
pip install transformers==4.51.3 accelerate
conda activate qwen
cd T2I-ReasonBench
```

<a name="prepare_images"></a>
## Prepare Evaluation Images

Generate images of your model using the T2I-ReasonBench prompts provided in the `prompts` directory. Organize them in the following structure for each dimension (using *idiom_interpretation* as an example):

```
../images/idiom_interpretation
├── 0001.png
├── 0002.png
├── 0003.png
├── 0004.png
...
└── 0200.png
```


<a name="mllm_eval"></a>
## MLLM-based Evaluation

###  Run the Evaluation Scripts

The following evaluation scripts have been placed in the `evaluation/Qwen2.5-VL/` directory:

- `eval_idiom.py`
- `eval_textual_image.py`
- `eval_entity.py`
- `eval_scientific.py`

Prepare the image repository path (*e.g.*, "../image/idiom_interpretation") in the argument `--image_folder`. 
Configure the folder to store the csv result files with the `--output_path` argument, 
configure the json file containing meta information with the `--qs_json` argument. 

<a name="idiom_interpretation"></a>
#### Example: Idiom Interpretation

Input the image folder and run the command:

```
python evaluation/Qwen2.5-VL/eval_idiom.py \
  --image_folder ../images/idiom_interpretation \
  --output_path csv_result/idiom \
  --prompt_json prompts/idiom_interpretation.json \
  --qs_json deepseek_evaluation_qs/evaluation_idiom.json \
  --model_name mymodel
```

The conversations with the MLLM will be saved in a CSV file: `csv_result/idiom/mymodel.csv`. 

The image id and prompt for each text-image pair will be recorded in the columns named of "id" and "prompt". 





