Diagnosing the Effects of Pre-training Data on Fine-tuning and Subgroup Robustness for Occupational NER in Clinical Notes
Track: regular paper (up to 6 pages)
Keywords: Robustness, Subgroups, NER, Finetuning, Occupation, Pre-training, Clinical Notes.
Abstract: This work evaluates Named Entity Recognition (NER) across five large language models (LLMs) using real-world narratives from healthcare and general-purpose datasets, focusing on occupational biases and cross-domain robustness. While prior studies have primarily examined biases in name-based entities using short sentence templates, we shift the focus to evaluating occupational NER in long note templates, analyzing biases across gender, race, and annual wage dimensions. Additionally, we assess cross-domain performance to understand how well the models generalize to unseen domain-specific data, such as healthcare datasets. Our evaluation demonstrates the effectiveness of fine-tuning on domain-specific datasets in improving performance compared to zero-shot and universal NER models. However, significant disparities in model performance and bias representation are observed, highlighting the need for targeted mitigation strategies to ensure subgroup robustness in real-world NER applications.
Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.
Format: Maybe: the presenting author will attend in person, contingent on other factors that still need to be determined (e.g., visa, funding).
Funding: Yes, the presenting author of this submission falls under ICLR’s funding aims, and funding would significantly impact their ability to attend the workshop in person.
Presenter: ~Dana_Moukheiber1
Submission Number: 58
Loading