Benchmark Dataset for Radiology Report Generation with Instructions and Contexts

Brian Nlong Zhao, Zilong Wang, XINYANG JIANG, Xufang Luo, Yifan Yang, Bo Li, Javier Alvarez-Valle, Matthew P. Lungren, Dongsheng Li, Lili Qiu

Keywords: Report Generation, Dataset and Benchmark, Multimodal Learning, Multimodal Model

TL;DR: New report generation task, data generation pipeline, new dataset in real-world clinical setting with medical contexts. A novel model architecture to adapt general domain multimodal LLM to medical domain and report generation task.

Abstract: While automatic report generation has demonstrated promising results using deep learning-based methods, deploying these algorithms in real-world scenarios remains challenging, where models may be required to follow the instruction from the radiologists and consider contextual information. Such instructional report generation tasks are critical for enabling more accurate, customizable, and scalable report generation processes, but remain under-explored and lack substantial datasets for training and evaluation. However, constructing a dataset for report generation with instructions and contexts is challenging due to the scarcity of medical data, privacy concerns and the absence of recorded user-model interactions. To tackle this challenge, we propose a unified and automatic data generation pipeline which leverages large language model (LLM) to produce high-quality instructions and context for report generation tasks. We present a new benchmark dataset MIMIC-R3G that extends the largest existing radiology report generation dataset MIMIC-CXR, comprising five representative tasks pertinent to real-world medical report generation. We conducted an extensive evaluation of state-of-the-art methods using the proposed benchmark datasets. Additionally, we introduced a baseline method, the Domain-enhanced Multimodal Model (DeMMo), demonstrating that leveraging training data containing instructions and contextual information significantly improves the performance of instructional report generation tasks.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8857