What Does Infect Mean to Cardio? Investigating the Role of Clinical Specialty Instructions in Medical LLMs
Abstract: In this paper, we introduce S-MedQA, a medical question-answering (QA) dataset for benchmarking large language models (LLMs) in fine-grained clinical specialties. Using S-MedQA, we gauge the role of instructions in knowledge-intense scenarios by checking the applicability of two popular hypotheses related to knowledge injection and style/format learning. We show that in the medical domain, more instructions result in better performance. However, the improvement in performance derives neither from the extra knowledge contained in the instructions nor from the style/format learned from them. Thus, we suggest rethinking the role of instruction data in the medical domain. We release S-MedQA for the community.
Paper Type: Short
Research Area: NLP Applications
Research Area Keywords: clinical NLP, healthcare applications
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources
Languages Studied: English
Submission Number: 3117
Loading