Are Large Language Models Good Word Sense Disambiguators?

ACL ARR 2024 December Submission127 Authors

Abstract:

Word Sense Disambiguation (WSD) is a historical task in computational linguistics that has received much attention over the years. However, with the advent of Large Language Models (LLMs), interest in this task (in its classical definition) has decreased.
In this study, we evaluate the performance of various LLMs on the WSD task. We extend a previous benchmark (XL-WSD) to re-design two subtasks suitable for LLM: 1) given a word in a sentence, the LLM must generate the correct definition; 2) given a word in a sentence and a set of predefined meanings, the LLM must select the correct one. The extended benchmark is built using the XL-WSD and BebelNet. The results indicate that LLMs perform well in zero-shot learning but cannot surpass current state-of-the-art methods. However, a fine-tuned model with a medium number of parameters outperforms all other models, including the state-of-the-art.

Paper Type: Long

Research Area: Semantics: Lexical and Sentence-Level

Research Area Keywords: polysemy, prompting, fine-tuning, automatic evaluation, multilingualism, few-shot generation

Contribution Types: Model analysis & interpretability, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English, Spanish, Italian, French, German

Submission Number: 127