An Evaluation of Large Language Models in Financial Sentiment Analysis

Alphaeus Dmonte, Eunmi Ko, Marcos Zampieri

Abstract: Financial sentiment analysis can help in understanding market trends and help organizations and individuals make important business decisions. Several machine learning approaches have been used for financial sentiment analysis over the years ranging from lexicon-based approaches to the use of deep neural networks and transformer-based models. Recent advances in Large Language Models (LLM) have prompted the use of these models for various Natural Language Processing (NLP) tasks, however, these models have not yet been substantially explored in the financial domain. In this paper, we evaluate the performance of various LLMs and we introduce a small benchmark dataset consisting of excerpts extracted from the Federal Reserve chair’s speeches. We use this dataset along with other existing datasets to evaluate LLMs using in-context learning approaches. We compare the F1 scores of these models with the state-of-the-art BERT-based models and analyze our results.