HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs

Anonymous

TL;DR: A unified one-stage protocol for domain adaptation and a SOTA Chinese medical LLM.

Abstract: Adapting a language model into a specific domain, a.k.a `domain adaption', is a common practice when specialized knowledge, e.g. medicine, is not encapsulated in a general language model like Llama2. This typically involves a two-stage process including continued pre-training and supervised fine-tuning. Implementing a pipeline solution with these two stages introduces additional complexity, particularly due to the challenge of managing dual data distribution shifts (i.e. firstly from general to domain-specific data and secondly from pre-training to fine-tuning data). To mitigate these, we propose a single-stage domain adaption protocol where heterogeneous data from both the pre-training and supervised stages are unified into a simple instruction-output pair format. Subsequently, a data priority sampling strategy is introduced to adaptively adjust data mixture during training. Following this protocol, we trained HuatuoGPT-II, a specialized LLM for the medical domain in Chinese. HuatuoGPT-II achieved state-of-the-art (SOTA) performance across multiple benchmarks, validating the efficacy of our one-stage protocol. The loss curve shows that the simplicity of the proposed training protocol improves training stability.

Paper Type: long

Research Area: NLP Applications

Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models

Languages Studied: Chinese, English