Semantic Homogeneity As Demonstration: Batch-Structured Semi-Supervised In-Context Learning for Natural Language Understanding

Cheng Chen, Yuangang Pan, Ivor Tsang

Keywords: In-Context Learning, Natural Language Understanding, Prompt Engineering / Prompting, Aggregate Ranking

Abstract:

In-context learning (ICL) adapts large language models (LLMs) to downstream natural language understanding (NLU) tasks by prepending a small set of labeled demonstrations (input--label exemplars) to each query. While effective, this paradigm is costly and fragile: curating representative demonstrations and maintaining their relevance at scale is difficult, and inference cost grows with prompt length. This motivates a complementary question: \emph{can LLMs benefit from in-context signals without using explicit exemplar pairs at all?} We propose \textbf{B}atch-Structured \textbf{I}mplicit \textbf{D}emonstration-Free \textbf{S}emi-supervised ICL (\textbf{BIDS}-ICL).
Instead of providing exemplar pairs, we use a small labeled seed set only to induce \emph{semantic structure}: we embed and cluster test-time inputs into \emph{semantically homogeneous batches}, then prompt the LLM with the batch as context for predicting the labels of all items in that batch.
In this non-exemplar regime, batch structure itself becomes an informative conditioning signal.
We further consider a practical extension that arises naturally from the clustering pipeline: each item may be accompanied by a \emph{pseudo-label hint} (e.g., an encoder-predicted intent), which can be noisy due to cluster mis-assignment and label propagation. Rather than asking whether pseudo-labels are universally good or bad, we ask a conditional question: \emph{when is it useful to expose an LLM to pseudo-label hints under batch-structured prompting?} On the theory side, we provide a Bayesian aggregation perspective and draw on stagewise Plackett--Luce (PL) aggregation to explain why semantically homogeneous batches can improve prediction reliability. Empirically, across eight datasets and two LLMs, we observe a consistent competency--homogeneity interaction: semantic homogeneity acts as an orthogonal in-context signal that systematically modulates pseudo-label utility. When batches exhibit low homogeneity, pseudo-label hints often amplify clustering noise and may underperform unlabeled structured batching. When homogeneity is high, pseudo-label hints become more reliable, though their marginal benefit diminishes when structural coherence alone already induces strong label separation.

Submission Number: 125