PatientAdvocateLM: A Clinically Grounded Retrieval-Augmented Conversational Agent for Medical Visit Preparation

Siyang Liu, Shruti Jayaraman, Lawrence An, Rada Mihalcea

Keywords: AI for Medical Applications, Retrieval-augumented Generation, AI-Assisted Clinical Tool

Abstract:

We explore an LLM-driven patient advocate to help patients clarify concerns and ask questions for their next medical visits. Given the high-stakes, patient-facing, and open-ended nature of the task, our study is highly exploratory in both design and evaluation.
We explore designs that structure free-form LLM interaction with a clinically validated framework and enhance information quality by retrieving content from vetted sources. We evaluate our system against a vanilla LLM through a within-subject study with ten human patients in the context of localized prostate cancer, assessing cognitive load during preparation.
We also perform an evaluation with 43 simulated patients using reference-based metrics.
Results from both evaluations complementarily show that our system can effectively help patients prepare for their medical visits.
Compared to a vanilla best-performing LLM, our system generates more relevant questions and reduces patients' cognitive load during preparation.
We also identify several design lessons to guide future refinement.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 52