Advancing EHR analysis: Predictive medication modeling using LLMs

Hanan M. Alghamdi, Abeer Mostafa

Abstract: In modern healthcare systems, the analysis of Electronic Health Records (EHR) is fundamental for uncovering patient health trends and enhancing clinical practices. This study aims to advance EHR analysis by developing predictive models for prescribed medication prediction using the MIMIC-IV dataset. We address data preparation challenges through comprehensive cleaning and feature selection, transforming structured patient health data into coherent sentences suitable for natural language processing (NLP). We fine-tuned several state-of-the-art large language models (LLMs), including Llama2, Llama3, Gemma, GPT-3.5 Turbo, Meditron, Claude 3.5-Sonnet, DeepSeek-R1, Falcon and Mistral, using tailored prompts derived from EHR data. The models were rigorously evaluated based on Cosine similarity, recall, precision, and F1-score, with BERTScore as the evaluation metric to address limitations of traditional n-gram-based metrics. BERTScore utilizes contextualized token embeddings for semantic similarity, providing a more accurate and human-aligned evaluation. Our findings demonstrate that the integration of advanced NLP techniques with detailed EHR data significantly improves medication management predictions. This research highlights the potential of LLMs in clinical settings and underscores the importance of seamless integration with EHR systems to improve patient safety and healthcare delivery.