3DS: Medical Domain Adaptation of LLMs via Decomposed Difficulty-based Data Selection

ACL ARR 2025 May Submission7840 Authors

Abstract: Large Language Models (LLMs) excel in general language tasks, motivating their adaptation to specialized domains such as healthcare. Effective domain adaptation typically involves supervised fine-tuning (SFT) on carefully selected instruction-tuning data. Current data selection methods adopt a data-centric approach, relying on external annotations and heuristics to identify external defined high-quality and challenging data. Our exploratory experiments highlight this approach fails to improve model's domain performance, due to misalignment between selected data and the model’s knowledge distribution. To tackle this, we propose Decomposed Difficulty-based Data Selection (3DS), a two-stage model-centric data selection framework that aligns data selection with the model’s distribution. 3DS employs a Prompt-Driven Data Selection to filter out noisy data based on the model's knowledge via explicit alignment in Stage#1, then adopts a Decomposed Difficulty-based Data Selection to guide selection via three novel data difficulty metrics, including Instruction Understanding, Response Confidence, and Response Correctness in Stage#2. These metrics are enhanced by an attention-based importance weighting mechanism for accurate calibration. Extensive experiments in the healthcare domain show 3DS outperforms existing methods by over 2.97% accuracy, with additional validation in the law domain confirming its generalization ability. Our dataset and code are open-sourced at https://anonymous.4open.science/r/3DS-E67F.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: healthcare applications, clinical NLP

Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Data resources

Languages Studied: Chinese, English

Submission Number: 7840