Developing healthcare language model embedding spaces

Niall Taylor, Dan Schofield, Andrey Kormilitzin, Dan W. Joyce, Alejo J. Nevado-Holgado

Abstract: Highlights•Healthcare meta data incorporated through contrastive loss pre-training of LLM.•Contrastive loss improves performance of frozen LLM on classification tasks.•UK healthcare datasets require specific domain adaptation in small LLMs.