Compression of Vision Transformer by Reduction of Kernel Complexity

ICLR 2026 Conference Submission20423 Authors

Keywords: Kernel Complexity, Channel Selection, Vision Transformer

TL;DR: We propose a new compact vision transformer architecture, the KCR-Transformer, which prunes the attention channels by reduction of a new and principled kernel complexity (KC) and renders compact models with competitive performance.

Abstract:

Self-attention and transformer architectures have become foundational components in modern deep learning. Recent efforts have integrated transformer blocks into compact neural architectures for computer vision, giving rise to various efficient vision transformers. In this work, we introduce Transformer with Kernel Complexity Reduction, or KCR-Transformer, a compact transformer block equipped with differentiable channel selection, guided by a novel and sharp theoretical generalization bound. To reduce the substantial computational cost of the MLP layers, the KCR-Transformer performs channel selection on the outputs of its self-attention layer.
Furthermore, we provide a rigorous theoretical analysis establishing a tight generalization bound for networks equipped with KCR-Transformer blocks. Leveraging such strong theoretical results, the channel pruning by KCR-Transformer is conducted in a generalization-aware manner, ensuring that the resulting network retains a provably small generalization error.
Our KCR-Transformer is compatible with many popular and compact transformer networks, such as ViT and Swin, and it reduces the FLOPs of the vision transformers while maintaining or even improving the prediction accuracy. In the experiments, we replace all the transformer blocks in the vision transformers with KCR-Transformer blocks, leading to KCR-Transformer networks with different backbones. The resulting KCR-Transformers achieve superior performance on various computer vision tasks, achieving even better performance than the original models with even less FLOPs and parameters. The code of the KCR-Transformer is available at \url{https://anonymous.4open.science/status/KCR-Transformer}.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 20423