Foresight v2 - A Large Language Model Medical Forecaster

Anonymous

Abstract:

Foresight v2 (FS2) is a Large Language Model based on LLaMa v2 7B and fine-tuned on hospital data for modelling patient timelines. It is capable of understanding a patient's clinical notes and forecasting SNOMED codes for a wide range of biomedical use cases including disorder prediction, medication recommendation, risk prediction, procedure recommendation and many more. FS2 is trained on the free text portion of the MIMIC-III dataset, firstly through the extraction of biomedical concepts and then the creation of contextualised patient timelines, upon which the model is then fine-tuned.
The results show significant improvement over the previous state-of-the-art for the next new biomedical concept prediction (P/R - 0.71/0.64 vs 0.52/0.32) and a similar improvement specifically for the next new disorder forecast (P/R - 0.66/0.59 vs 0.46/0.25). Finally, on the task of disorder forecast, we compare this model, to GPT-4-turbo, and show that FS2 performs significantly better on such tasks (P@5 - 0.84 vs 0.62). This highlights the need to incorporate real health data into LLMs and shows that even much smaller models when fine-tuned on high-quality specialised data outperform much larger ones.

Paper Type: long

Research Area: NLP Applications

Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models

Languages Studied: English

Preprint Status: There is no non-anonymous preprint and we do not intend to release one.

A1: yes

A1 Elaboration For Yes Or No: 4.1

A2: yes

A2 Elaboration For Yes Or No: 4.1

A3: yes

A3 Elaboration For Yes Or No: 1

B: yes

B1: yes

B1 Elaboration For Yes Or No: 1

B2: yes

B2 Elaboration For Yes Or No: 4.2

B3: yes

B3 Elaboration For Yes Or No: 4.2

B4: n/a

B5: yes

B5 Elaboration For Yes Or No: 3

B6: yes

B6 Elaboration For Yes Or No: 3

C: yes

C1: yes

C1 Elaboration For Yes Or No: 2.2

C2: yes

C2 Elaboration For Yes Or No: 2.2

C3: yes

C3 Elaboration For Yes Or No: 3

C4: yes

C4 Elaboration For Yes Or No: Available in the open sourced code

D: no

D1: n/a

D2: n/a

D3: n/a

D4: n/a

D5: n/a

E: yes

E1: yes

E1 Elaboration For Yes Or No: Our model was compared to GPT-4-turbo, it is covered in multiple sections in the paper - but mainly in section 3 Results