The Silent Vote: Improving Zero-Shot LLM Classification by Aggregating Semantic Neighborhoods

ACL ARR 2026 January Submission2426 Authors

Keywords: Zero-Shot Classification, Large Language Models, Model Calibration, Aleatoric Uncertainty, Constrained Decoding, Expected Calibration Error (ECE), Human Alignment, Toxicity Detection

Abstract:

Large Language Models are increasingly used as zero-shot classifiers in complex reasoning tasks. However, standard constrained decoding suffers from a phenomenon we define as Renormalization Bias. When a model is restricted to a small set of target labels, the standard softmax operation discards the probability mass assigned to semantic synonyms in the original distribution. This loss of information, which we call the Silent Vote, results in artificial overconfidence and poor calibration.

We propose Semantic Softmax, an inference-time layer that recovers this lost information by aggregating the scores of the semantic neighborhood surrounding each target label. We evaluate our approach using Qwen-2.5-1.5B and Phi-4-mini on the GoEmotions and Civil Comments datasets. Our results demonstrate consistent improvements across all evaluation metrics: Semantic Softmax substantially reduces Expected Calibration Error (ECE) and Brier Score, while simultaneously enhancing discriminative performance in terms of AUROC and Macro-F1. By accounting for linguistic nuances, our method provides a more calibrated and accurate alternative for zero-shot classification.

Paper Type: Short

Research Area: Machine Learning for NLP

Research Area Keywords: structured prediction; word embeddings

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 2426