Position: Thematic Analysis of Unstructured Clinical Transcripts with Large Language Models

Seungjun Yi, Joakim Nguyen, Terence Lim, Andrew Well, Joseph Skrovan, Mehak Beri, YongGeon Lee, Kavita Radhakrishnan, Liu Leqi, Mia Markey, Ying Ding

Keywords: Large Language Models, Thematic Analysis, Clinical Transcripts, Qualitative Research, Evaluation Framework, Healthcare NLP

TL;DR: We review LLM use in thematic analysis of clinical transcripts, highlight fragmented methods and evaluation, and propose a standardized framework based on validity, reliability, and interpretability.

Abstract:

This position paper examines how large language models (LLMs) can support
thematic analysis of unstructured clinical transcripts, a widely used but resource-
intensive method for uncovering patterns in patient and provider narratives. We
conducted a systematic review of recent studies applying LLMs to thematic analysis,
complemented by an interview with a practicing clinician. Our findings reveal
that current approaches remain fragmented across multiple dimensions including
types of thematic analysis, datasets, prompting strategies and models used, most
notably in evaluation. Existing evaluation methods vary widely (from qualitative
expert review to automatic similarity metrics), hindering progress and preventing
meaningful benchmarking across studies. We argue that establishing standardized
evaluation practices is critical for advancing the field. To this end, we propose
an evaluation framework centered on three dimensions: validity, reliability, and
interpretability.

Submission Number: 82