Next Token Perception Score: Analytical Assessment of your LLM Perception Skills

ICLR 2026 Conference Submission13538 Authors

Keywords: autoregressive pretraining; feature subspace alignment; linear probe; low-rank adaptation; large language models

TL;DR: The paper introduces the Next Token Perception Score (NTPS), a closed-form metric that quantifies how well pretrained LLM representations align with downstream perception tasks, strongly predicting probe accuracy and fine-tuning gains.

Abstract: Autoregressive pretraining has become the de facto paradigm for learning general-purpose representations in large language models (LLMs). However, linear probe performance across downstream perception tasks (e.g., classification, regression) shows substantial variability, suggesting that features optimized for next-token prediction do not consistently transfer well to downstream perception tasks. We demonstrate that representations learned via autoregression capture features that may lie outside the subspaces most informative for perception. To quantify the (mis)alignment between autoregressive pretraining and downstream perception, we introduce the Next Token Perception Score (NTPS), a score derived under a linear setting that measures the overlap between autoregressive and perception feature subspaces. This metric can be efficiently computed in closed form from pretrained representations and labeled data, and is proven to both upper- and lower-bound the excess loss. Empirically, we show that NTPS correlates strongly with linear probe accuracy across 12 diverse NLP datasets and eight pretrained models ranging from 270M to 8B parameters, confirming its utility as a measure of alignment. Additionally, NTPS reliably predicts the additional accuracy gains attained by LoRA finetuning thereby providing a lightweight prescreening tool for LoRA adaptation. Our results offer both theoretical insights and practical tools for analytically assessing LLM perception skills.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 13538