ICLEval: Evaluating In-Context Learning Ability of Large Language Models

Anonymous

Abstract: In-Context Learning (ICL) is a crucial capability of LLMs as it enables them to understand and reason across a series of interconnected inputs. However, existing evaluation frameworks primarily focus on language abilities and knowledge, often neglecting the evaluation of ICL ability. This limitation hampers our understanding of how LLMs utilize context in complex problem-solving. In this study, we introduce the ICLEval benchmark to assess the ICL abilities of LLMs systematically. We evaluate two fundamental abilities: copying and learning. We also investigate the impact of model size, pretraining stage, and other factors on ICL abilities. Our findings reveal that model size is not the sole determinant of ICL efficacy. Surprisingly, we observe that ICL abilities, particularly copying, develop early in the pretraining process and stabilize afterward. Furthermore, we discover that the ICL abilities also be influenced by some other factors, such as distinguishing ability, inherent preferences, attention points capacity, and tokenizer.

Paper Type: long

Research Area: Resources and Evaluation

Contribution Types: Data resources, Data analysis

Languages Studied: English