Instruction-guided deidentification with synthetic test cases for Norwegian clinical text

Jørgen Aarmo Lund, Karl Øyvind Mikalsen, Joel Burman, Ashenafi Zebene Woldaregay, Robert Jenssen

Keywords: deidentification anonymization pseudonymization nlp ner named-entity-recognition instruction prompt llm healthcare privacy gpt gpt4 medicine clinical

TL;DR: Presents a synthetic dataset for deidentification of Norwegian clinical text, and proposes and tests methods for performing deidentification with instruction-tuned language models

Abstract: Deidentification methods, which remove directly identifying information, can be useful tools to mitigate the privacy risks associated with sharing healthcare data. However, benchmarks to evaluate deidentification methods are themselves often derived from real clinical data, making them sensitive themselves and therefore harder to share and apply. Given the rapid advances in generative language modelling, we would like to leverage large language models to construct freely available deidentification benchmarks, and to assist in the deidentification process. We apply the GPT-4 language model to, for the first time, construct a synthetic and publicly available dataset of synthetic Norwegian discharge summaries with annotated identifying details, consisting of 1200 summaries averaging 100 words each. In our sample of documents, we find that the generated annotations highly agree with human annotations, with an $F_1$ score of $0.983$. We then examine whether large language models can be applied directly to perform deidentification themselves, proposing methods where an instruction-tuned language model is prompted to either annotate or redact identifying details. Comparing the methods on our synthetic dataset and the NorSynthClinical-PHI dataset, we find that GPT-4 underperforms the baseline method proposed by Bråthen et al. (2021), suggesting that named entity recognition problems are still challenging for instruction-tuned language models.

Git: https://github.com/UNN-SPKI/Nor-DeID-SynthData

Project: https://github.com/UNN-SPKI/Nor-DeID-SynthData

Submission Number: 46