On-the-fly Definition Augmentation of LLMs for Biomedical NER

Anonymous

Abstract: Despite their general success, LLMs still lag behind on biomedical named entity recognition (NER) tasks, which are difficult due to the presence of specialized terminology and lack of training data. In this work we set out to improve LLM performance on biomedical NER in limited data settings via: (i) A new knowledge augmentation approach which incorporates definitions of relevant concepts on-the-fly, and (ii) A comprehensive exploration of prompting strategies. Our experiments show that the proposed definition augmentation approach is useful for both open source and closed LLMs. For example, it increases GPT-4 performance (F1) by 15% on average across all (six) of our test datasets. We conduct extensive ablations and analyses to demonstrate that these performance improvements stem from adding relevant knowledge about definitions. We find that careful prompting strategies also improve LLM scores, allowing them to outperform fine-tuned language models in few-shot settings. To facilitate future research in this direction, we plan to release our code upon acceptance.

Paper Type: long

Research Area: Information Extraction

Contribution Types: NLP engineering experiment, Approaches to low-resource settings

Languages Studied: English