Assessing the Efficacy of Pre-Trained and Large Language Models for Health Classification with Varying Data Volumes

ACL ARR 2025 February Submission2486 Authors

Abstract:

Automated text classification in medical and health domains enables the extraction of structured information from unstructured clinical text, such as identifying diseases and associated conditions.
However, applying text classification models effectively in healthcare requires a nuanced understanding of specific subtopics and the trade-offs between model scale and available data resources.
This paper evaluates the performance of pretrained language models (PLMs) and large language models (LLMs) in classifying subtopics within the sleep and activity domains. Using a dataset of curated Reddit posts, we examine how classifier performance varies with different training sample sizes, including low-resource scenarios with just one to five examples. Our findings highlight a complex interaction between model architecture, data availability, and classification performance, demonstrating the strengths of LLMs in zero-shot learning in nuanced subdomains with limited data, while PLMs surpass LLMs with modest increases in data.
This research provides valuable insights into the optimal application of language models for health-related text classification tasks, especially under varying resource constraints.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: healthcare applications, clinical NLP, data-efficient training, zero/few-shot extraction

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: English

Submission Number: 2486