Are Llamas Sesquipedalian? Analyzing Rare Words in Large Language Models

Anonymous

Abstract: Large language models (LLMs) have changed the modern landscape of natural language processing (NLP). Due to their strong performance on multiple tasks, analyzing LLM performance in unusual or difficult scenarios is important. In this work, we investigate LLaMA's performance when using rare and unknown words, something previous transformer based models have been shown to struggle with. We apply various rare word experiments on Large Language Models, specifically LLaMA 7B and 13B. We demonstrate that LLMs still perform worse processing rare and unknown words compared to frequent words, but show that in contextualized scenarios, LLMs face far less deterioration using rare words than previous models.

Paper Type: short

Research Area: Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability

Languages Studied: English

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.