Exploring the Discriminative Capability of LLMs in In-context Learning

Hongduan Tian, Feng Liu, Tongliang Liu, Gang Niu, Chengqi Zhang, Bo Han

Keywords: large language models, in-context learning, discriminative capability, decision boundary

Abstract: _In-context learning_ (ICL), as an emergent behavior of large language models (LLMs), has exhibited impressive capability in solving previously unseen tasks based on the observations of the given samples without extra training. However, recent works find that LLMs irregularly obtain unexpected fragmented decision boundaries in simple discriminative tasks, such as binary linear classification. Our observations on the output of Llama-3-8B for the reasoning process of label predictions reveal that LLMs tend to leverage the existing machine learning algorithms to perform discriminative tasks. Specifically, LLMs tend first to select a strategy for the given task and then predict the labels of query data by executing the selected strategy. Based on the observation, in this paper, we propose to dive into such a behavior of LLMs for a deeper understanding of the discriminative capability of LLMs. We conduct a series of analyses on Llama-3-8B to determine the behaviors adopted by LLMs in the discriminative tasks, including probing the label predictions of query data and the corresponding confidence of LLMs under different prompt settings. Moreover, we also probe the preference of LLMs for strategy selection and then simulate the behavior of LLMs performing classification based on obtained preference. The analysis and simulation results provide some important observations and insights into the properties of LLMs in performing discriminative tasks.

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6046