Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model

Anonymous

Abstract: Integrating large language models (LLMs) into healthcare holds great potential but faces challenges. Pre-training LLMs from scratch for domains like medicine is resource-heavy and often unfeasible. On the other hand, sole reliance on Supervised Fine-tuning (SFT) can result in overconfident predictions and may not tap into domain-specific insights. In response, we present a multi-stage training method combining Domain-specific Continued Pre-training (DCPT), SFT, and Direct Preference Optimization (DPO). In addition, we publish a 3Gb Chinese Medicine (ChiMed) dataset, encompassing medical question answering, plain texts, knowledge graphs, and dialogues, segmented into three training stages. The medical LLM trained with our pipeline, Qilin-Med, shows substantial performance improvement. In the CPT and SFT phases, Qilin-Med achieved 38.4% and 40.0% accuracy on the CMExam test set, respectively. It outperformed the basemodel Baichuan-7B (accuracy: 33.5%), by 7.5%. In the DPO phase, it scored 16.66 in BLEU-1 and 27.44 in ROUGE-1 on the Huatuo-26M test set, bringing further improvement to the SFT phase (12.69 in BLEU-1 and 24.21 in ROUGE-1). Additionally, we have further enhanced the model's performance through the Retrieval Augmented Generation (RAG) approach. Experiments demonstrate that Qilin-Med-RAG achieves an accuracy rate of 42.8% on CMExam. These results highlight the contribution of our novel training approach in building LLMs for medical applications.

Paper Type: long

Research Area: Generation

Contribution Types: Publicly available software and/or pre-trained models, Data resources

Languages Studied: Chinese

Preprint Status: There is a non-anonymous preprint (URL specified in the next question).

A1: yes

A1 Elaboration For Yes Or No: See Section.6

A2: yes

A2 Elaboration For Yes Or No: See Section.7

A3: yes

A3 Elaboration For Yes Or No: See Section.1

B: yes

B1: yes

B1 Elaboration For Yes Or No: See Section.2

B2: yes

B2 Elaboration For Yes Or No: See Section.7

B3: yes

B3 Elaboration For Yes Or No: See Section.7

B4: yes

B4 Elaboration For Yes Or No: See Section.7

B5: n/a

B6: yes

B6 Elaboration For Yes Or No: See Section.3

C: yes

C1: yes

C1 Elaboration For Yes Or No: See Section.4.2

C2: yes

C2 Elaboration For Yes Or No: See Section.4.2

C3: yes

C3 Elaboration For Yes Or No: See Section.4

C4: yes

C4 Elaboration For Yes Or No: See Section.4

D: no

E: no