Lexicon Annotation with LLM: A Proof of Concept with ChatGPT

Francisco Supino Marcondes, Adelino de C. O. S. Gala, Manuel Rodrigues, José João Almeida, Paulo Novais

Abstract: Lexicon annotation is a critical yet time-consuming task that can hold back the progress of language-intensive projects. This paper explores the potential of Large Language Models (LLMs) to automate lexicon annotation, traditionally performed by humans. We present a proof of concept by evaluating ChatGPT's performance on annotating VADER's sentiment lexicon. Our findings demonstrate that ChatGPT achieves fair performance in this task, suggesting that LLMs can operate as a valuable tool for initial annotations, with subsequent refinements by domain specialists. This approach could significantly accelerate lexicon development and maintenance while balancing efficiency and accuracy. Our study provides insights into the capabilities and limitations of LLMs in lexicon annotation, leading the way for further research in automating linguistic resources development.