User-Centric Evaluation of LLM-Based Pseudonymization in Medical Texts

Stig Hellemans, Kris Laukens, Pieter Meysman

Track: Type E (Late-Breaking Abstracts)

Keywords: Natural Language Processing, Privacy preservation, Evaluation Frameworks

Abstract:

Protecting patient privacy while enabling data-driven research remains a central challenge in clinical informatics. De-identification methods such as pseudonymization, where identifiers are replaced by re-linkable pseudonyms under strict conditions, are widely used, yet training and evaluating such systems require extensive manual annotation. Large language models (LLMs) offer a promising way to reduce this burden by generating pre-annotations that human annotators only need to refine. To quantify the true efficiency gains, we introduce a user-centric evaluation metric inspired by annotation edit distance, which measures the number of user actions (additions, edits, deletions) required to reach gold-standard quality. This approach captures practical annotation workload more realistically than traditional precision or recall metrics.

We evaluated two grammar-based LLM anonymization methods on a corpus of 100 synthetic clinical narratives containing realistic protected health information (PHI). The Finite-State Machine (FSM) approach achieved the highest accuracy by wrapping sensitive spans directly in text but required full text rewriting, making it less token-efficient. The dictionary-based JSON grammar combined fuzzy matching with structured outputs compatible with all OpenAI APIs, offering a more lightweight alternative. Using our user-based metric and empirically measured action durations, the gemma-27b-it-q8 model paired with the FSM grammar reduced the number of required annotation actions by up to tenfold compared to a no-preannotation baseline, indicating substantial potential for human–AI collaboration in clinical text anonymization.

Beyond entity-level anonymization, we assessed pseudonymization robustness using Bayesian inference to analyze how temporal cues, such as weekdays or reference events (“New Year’s Eve”), may leak information even after date-shifting. By simulating adversarial inference, we quantified how quickly offset estimates converge toward true values, revealing that poorly designed shifting schemes can expose substantial temporal information. These findings highlight the need for rigorous adversarial testing of de-identification systems and for extending structured pseudonymization principles (e.g., the Shift and Truncate method) to unstructured text, where indirect temporal and linguistic cues remain challenging.

Overall, our results emphasize that effective anonymization must balance privacy protection and data utility: removing identifiers without erasing clinically relevant context. Grammar-constrained LLMs can serve as efficient assistants for pre-annotation, while user-based metrics provide a more realistic measure of annotation effort. Together, these methods advance the development of practical, verifiable, and human-centered anonymization pipelines for privacy-preserving clinical research.

Submission Number: 87