Medal Matters: Probing LLMs' Knowledge Structures Through Olympic Rankings

ACL ARR 2024 December Submission1734 Authors

Abstract: Large language models (LLMs) have achieved remarkable success in natural language processing tasks, yet their internal knowledge structures remain poorly understood. This study examines these structures through the lens of historical Olympic medal tallies, evaluating LLMs on two tasks: (1) retrieving medal counts for specific teams and (2) identifying rankings of each team. While state-of-the-art LLMs excel in reporting medal counts, they struggle with inferring rankings, highlighting a key difference between their knowledge organization and human reasoning. These findings shed light on the limitations of LLMs’ internal knowledge integration and suggest directions for improvement. To facilitate further research, we release our code, dataset, and model outputs.

Paper Type: Short

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: knowledge discovering, probing

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 1734