Medical Knowledge Adaptation for Large Language Models: Strategies and Comparative Analysis

ACL ARR 2025 February Submission4609 Authors

Abstract: The development of Large Language Models (LLMs) has led to increased focus on their adaptation to specialized domains and languages, particularly in settings with limited domain-specific data. While recent studies have questioned the benefits of domain-adaptive pre-training (DAPT) in English medical contexts, our work demonstrates that domain adaptation can be effective when strategically implemented. Using French medical domain adaptation as a case study, we systematically evaluate different adaptation strategies: continual pre-training (CPT), supervised fine-tuning (SFT), and combined approaches (CPT followed by SFT). Our study highlights that adapting a general-purpose model with novel domain data leads to significant gains (87\% win rate), whereas further adapting models already exposed to similar knowledge offers limited benefits. Moreover, while CPT+SFT achieves the best overall performance, direct SFT emerges as a strong, more computationally efficient alternative.

Paper Type: Long

Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond

Research Area Keywords: pre-training, continual learning, fine-tuning, applications, prompting

Contribution Types: Model analysis & interpretability, Approaches to low-resource settings, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models, Data resources

Languages Studied: French, English

Submission Number: 4609