Who Does the Model Think You Are? LLMs Exhibit Implicit Bias in Inferring Patients' Identities from Clinical Conversations

ACL ARR 2025 May Submission3612 Authors

Abstract:

Large Language Models (LLMs) are now established as powerful instruments for clinical decision-making, with rapidly growing applications across healthcare domains. Nevertheless, the presence of biases remains a critical barrier to their responsible deployment in clinical practice.
In this study, we develop a framework to systematically investigate implicit biases in LLMs within healthcare contexts, specifically focusing on doctor–patient conversations.
We study whether inclusion of relevant stereotypes or toxic remarks into de-identified clinical conversations can influence an LLM's demographic inferences --- in particular, prediction of the patient's gender and race.
Through empirical evaluation with state-of-the-art LLMs, including GPT-4o and Llama-3-70B, our findings demonstrate that LLMs exhibit major disparities.
Moreover, inclusion of stereotypical content can substantially influence the LLM's prediction of the patient's information, thereby underscoring the susceptibility of LLMs to stereotypes in clinical settings.
Additionally, a qualitative analysis on occasional model reasoning that accompany these predictions reveals insightful gender-specific associations.

Paper Type: Long

Research Area: Ethics, Bias, and Fairness

Research Area Keywords: Bias, fairness, toxicity in LLMs; implicit biases in clinical applications

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 3612