<div align="center">

  <h2><b> Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual Reasoning from Transit Maps </b></h2>
  <h4> A Fine-Grained Visual Reasoning Benchmark: ReasonMap</h4>

</div>


<div align="center">


</div>

<div align="center">


</div>


---

## Usage

### 1. Install dependencies

If you face any issues with the installation, please feel free to open an issue. We will try our best to help you.

```bash
conda create -n reasonmap python==3.10
pip install -r requirements.txt
```

### 2. Download the dataset
You can download the dataset from [HuggingFace](https://huggingface.co/datasets/AnonymousReasonMap/ReasonMap).


### 3. Evaluation
You can evaluate the model performance on ReasonMap by running the following command:
```bash
# open-source models
bash script/run.sh
# closed-source models
bash script/run-closed-models.sh

# after running the above scripts, you can analyze the results by:
python cal_metrics.py
```

## Citation
If you find this survey useful in your research, please consider citing our paper
