Text2MDT: Extracting Decision Trees from Medical Texts Using Large Language Models

Anonymous

Abstract: Knowledge of the medical decision process, which can be modeled as medical decision trees (MDTs), is critical to building clinical decision support systems. However, the current MDT construction methods rely heavily on time-consuming and laborious manual annotation. In this work, we propose a novel task, Text2MDT, to explore the automatic extraction of MDTs from medical texts such as medical guidelines and textbooks. We normalized the form of the MDT and created an annotated Text2MDT dataset in Chinese with the participation of medical experts. We investigate two different methods for the Text2MDT tasks: (a) an end-to-end framework that only relies on a GPT style large language models (LLM) instruction tuning to generate all the node information and tree structures. (b) The pipeline framework decomposes the Text2MDT task into three subtasks. Experiments on our Text2MDT dataset demonstrate that (a) the end-to-end method based on LLMs (7B parameters or larger) shows promising results and successfully outperforms the pipeline methods. (b) The chain-of-thought (COT) prompting method \cite{Wei2022ChainOT} can improve the performance of the fine-tuned LLMs on the Text2MDT test set. (c) the lightweight pipelined method based on encoder-based pre-trained models also performs well with LLMs with model complexity two magnitudes smaller.\footnote{Our Text2MDT dataset and the source codes are open-sourced, and we will make the dataset and the source codes openly available upon acceptance. }.

Paper Type: long

Research Area: NLP Applications

Contribution Types: NLP engineering experiment, Data resources

Languages Studied: Chinese

Preprint Status: There is a non-anonymous preprint (URL specified in the next question).

A1: yes

A1 Elaboration For Yes Or No: "Limitations" section

A2: yes

A2 Elaboration For Yes Or No: "Ethics Statement" section

A3: yes

A3 Elaboration For Yes Or No: "Abstract" and Section 1

B: yes

B1: yes

B1 Elaboration For Yes Or No: Section 3

B2: yes

B2 Elaboration For Yes Or No: Section 3

B3: yes

B3 Elaboration For Yes Or No: Section 3

B4: yes

B4 Elaboration For Yes Or No: Section 3

B5: yes

B5 Elaboration For Yes Or No: Section 3

B6: yes

B6 Elaboration For Yes Or No: Section 3

C: yes

C1: yes

C1 Elaboration For Yes Or No: Section 4

C2: yes

C2 Elaboration For Yes Or No: Section 4

C3: yes

C3 Elaboration For Yes Or No: Section 4

C4: yes

C4 Elaboration For Yes Or No: Section 4

D: yes

D1: yes

D1 Elaboration For Yes Or No: Appendix B

D2: yes

D2 Elaboration For Yes Or No: Appendix B

D3: yes

D3 Elaboration For Yes Or No: Appendix B

D4: yes

D4 Elaboration For Yes Or No: Appendix B

D5: yes

D5 Elaboration For Yes Or No: Appendix B

E: no

E1: n/a