Structuring Radiology Reports: Challenging LLMs with Lightweight Models

ACL ARR 2025 February Submission1213 Authors

Abstract:

Radiology reports are critical for clinical decision-making but often lack a standardized format, limiting both human interpretability and machine learning (ML) applications. While large language models (LLMs) like GPT-4 can effectively reformat these reports, their proprietary nature, computational demands, and data privacy concerns limit clinical deployment. To address this challenge, we employed lightweight encoder-decoder models (<300M parameters), specifically T5 and BERT2BERT, to structure radiology reports from the MIMIC-CXR and CheXpert databases. We benchmarked our lightweight models against five open-source LLMs (3-8B parameters), which we adapted using in-context learning (ICL) and low-rank adaptation (LoRA) finetuning. We found that our best-performing lightweight model outperforms all ICL-adapted LLMs on a human-annotated test set across all metrics (BLEU: 212\%, ROUGE-L: 63\%, BERTScore: 59\%, F1-RadGraph: 47\%, GREEN: 27\%, F1-SRRG-Bert: 43\%).
While the overall best-performing LLM (Mistral-7B with LoRA) achieved a marginal 0.3\% improvement in GREEN Score over the lightweight model, this required $10\times$ more training and inference time, resulting in a significant increase in computational costs and carbon emissions. Our results highlight the advantages of lightweight models for sustainable and efficient deployment in resource-constrained clinical settings.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: healthcare applications, clinical NLP, data augmentation

Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English

Submission Number: 1213