Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More

Feng Wang, Yaodong Yu, Wei Shao, Yuyin Zhou, Alan Yuille, Cihang Xie

Abstract: Since the introduction of Vision Transformer (ViT), patchification has long been regarded as a common image pre-processing approach for plain visual architectures. By compressing the spatial size of images, this approach can effectively shorten the token sequence and reduce the computational cost of ViT-like plain architectures. In this work, we aim to thoroughly examine the information loss caused by this patchification-based compressive encoding paradigm and how it affects visual understanding. We conduct extensive patch size scaling experiments and excitedly observe an intriguing scaling law in patchification: the models can consistently benefit from decreased patch sizes and attain improved predictive performance, until it reaches the minimum patch size of 1*1, i.e., pixel tokenization. This conclusion is broadly applicable across different vision tasks, various input scales, and diverse architectures such as ViT and the recent Mamba models. Moreover, as a by-product, we discover that with smaller patches, task-specific decoder heads become less critical for dense prediction. In the experiments, we successfully scale up the visual sequence to an exceptional length of 50,176 tokens, achieving a competitive test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We hope this study can provide insights and theoretical foundations for future works of building non-compressive vision models.

Lay Summary: In modern computer vision, images are usually split into small patches before being analyzed by AI models. This makes computation faster but may lose important details. In our work, we investigate how much this “patchification” affects model performance. By using smaller and smaller patches—down to individual pixels—we find that models actually perform better across various tasks like image classification and object detection. Surprisingly, removing the patch step entirely and treating every pixel as its own token allows the model to reach top accuracy, even with very long input sequences. Our findings show that it's possible to move away from compressive methods and build more accurate models by simply using all the raw visual information.

Link To Code: https://github.com/wangf3014/Patch_Scaling.

Primary Area: Applications->Computer Vision

Keywords: Scaling law, Patchification

Submission Number: 2689