Evaluating Large Language Models in an Emerging Domain: A Pilot Study in Decentralized Finance

Joshua Carter Pearlson, Xiaoyuan Liu, Chengsong Huang, Kripa Ann George, Dawn Song, Chenguang Wang

Keywords: NLP, DEFI, NER

Abstract: We introduce a new challenge to test the emergent abilities of large language models. Unlike standard benchmarks that aim to examine the ability with existing domains, we propose to test it in a new domain, decentralized finance (DeFi). DeFi has the potential to rewire how the financial system works. Growing to over $250 billion within three years of existence, DeFi's growth is rapid and unprecedented. This domain presents a natural testbed for emergent abilities. A large number of new concepts and entities such as specific cryptocurrencies were released after models stopped training. We create the first dataset resource in this domain with high-quality manual annotations, with a focus on named entity recognition. Our results show, while state of the art models produce reasonable performance in recognizing entities that already existed before they completed training, their performance drops dramatically when new entities are presented. Although improved performance is obtained through teaching models on the training portion of our dataset, the results suggest fundamental algorithmic innovations are required to equip models with emergent intelligence.

Submission Number: 40