Enabling Scalable Evaluation of Bias Patterns in Medical LLMs
Keywords: Medical LLMs, Fairness Evaluation
TL;DR: We present a method to generate evidence-based clinical scenarios (vignettes) for evaluating bias patterns in medical LLMs at large scale.
Abstract: Large language models (LLMs) have shown impressive potential in helping with numerous medical challenges. Deploying LLMs in high-stakes applications such as medicine, however, brings in many concerns. One major area of concern relates to biased behaviors of LLMs in medical applications, leading to unfair treatment of individuals. To pave the way for the responsible and impactful deployment of Med LLMs, rigorous evaluation is a key prerequisite. Due to the huge complexity and variability of different medical scenarios, existing work in this domain has primarily relied on using manually crafted datasets for bias evaluation. In this study, we present a new method to scale up such bias evaluations by automatically generating test cases based on rigorous medical evidence. We specifically target the challenges of domain-specificity of bias characterization, hallucinating while generating the test cases, and various dependencies between the health outcomes and sensitive attributes. To that end, we offer new methods to address these challenges integrated with our generative pipeline. Specifically, we use medical knowledge graphs and medical ontologies; and customize general LLM evaluation frameworks in our method. Through a series of extensive experiments, we show that the test cases generated by our proposed method are reliable and can effectively reveal bias patterns in LLMs. Additionally, we publish a large bias evaluation dataset, which provides a comprehensive platform for testing and improving the fairness of clinical LLMs. A live demo of our application for vignette generation is available at https://vignette.streamlit.app. Our code is also available at https://anonymous.4open.science/r/vignette_llm-2853.
Primary Area: alignment, fairness, safety, privacy, and societal considerations
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3814
Loading