LLM-based Solutions for Healthcare Chatbots: a Comparative Analysis

Sara Montagna, Gianluca Aguzzi, Stefano Ferretti, Martino Francesco Pengo, Lorenz Cuno Klopfenstein, Michelangelo Ungolo, Matteo Magnini

Abstract: This paper discusses the challenges of using Large Language Models (LLMs) in medical chatbots for chronic disease self-management. Accordingly, we define an architecture specifically devised to deal with issues related to reliability, clinical trials, and privacy. Two solutions are compared to prevent data disclosure: a filtering mechanism for sensitive data with an external LLM, and a locally deployed LLM using open-source models. Experimental results underscore the challenges in effectively instructing the local LLM so as to provide performances comparable to GPT-3.5.