What Does Infect Mean to Cardio? Investigating the Role of Clinical Specialty Instructions in Medical LLMs

ACL ARR 2025 February Submission5188 Authors

Abstract:

In this paper, we introduce S-MedQA, an English medical question-answering (QA) dataset for benchmarking large language models in fine-grained clinical specialties. We use S-MedQA to
check the applicability of a popular hypothesis related to knowledge injection in the knowledge-intense scenario of medical QA, and show
1) that training on data from a speciality does not necessarily lead to best performance on that specialty and
2) regardless of the specialty fine-tuned on, token probabilities of clinically relevant terms for all specialties increase consistently.

Paper Type: Short

Research Area: NLP Applications

Research Area Keywords: clinical NLP, healthcare applications

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources

Languages Studied: English

Submission Number: 5188