Revisiting [CLS] and Patch Token Interaction in Vision Transformers

ICLR 2026 Conference Submission5894 Authors

Keywords: representation, vision, transformer, SSL, attention, specialization, architecture, interpretability, DINO, DINOv2, CLIP, DEIT

TL;DR: We propose and analyze a new architecture to specialize CLS and patch tokens processing in ViTs, enhancing dense tasks performances.

Abstract:

Vision Transformers have emerged as powerful, scalable and versatile representation learners. To capture both global and local features, a learnable [CLS] class token is typically prepended to the input sequence of patch tokens. Despite their distinct nature, both token types are processed identically throughout the model.
In this work, we investigate the friction between global and local feature learning under different pre-training strategies by analyzing the interactions between class and patch tokens.
Our analysis reveals that standard normalization layers introduce an implicit differentiation between these token types. Building on this insight, we propose specialized processing paths that selectively disentangle the computational flow of class and patch tokens, particularly within normalization layers and early query-key-value projections.
This targeted specialization leads to significantly improved patch representation quality for dense prediction tasks. Our experiments demonstrate segmentation performance gains of over 2 mIoU points on standard benchmarks, while maintaining strong classification accuracy. The proposed modifications introduce only an 8\% increase in parameters, with no additional computational overhead.
Through comprehensive ablations, we provide insights into which architectural components benefit most from specialization and how our approach generalizes across model scales and learning frameworks.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 5894