Efficiency at scale: Investigating the performance of diminutive language models in clinical tasks

Niall Taylor, Upamanyu Ghose, Omid Rohanian, Mohammadmahdi Nouriborji, Andrey Kormilitzin, David A. Clifton, Alejo J. Nevado-Holgado

Abstract: Highlights•State of the art performance in Clinical NLP using efficient fine-tuning methods.•25 million parameter LLMs benefit from LoRA fine-tuning.•Classification performance matched with 98% fewer trained parameters.•Trade-off in performance in tiny LLMs outweighs cost of much larger Language Models.