Pyramid Tokens-to-Token Vision Transformer for Thyroid Pathology Image Classification

Peng Yin, Bo Yu, Chengwei Jiang, Hechang Chen

Abstract: Histopathological image contains rich phenotypic information, which is beneficial to classifying tumor subtypes and predicting the development of diseases. The vast size of pathological slides makes it impossible to directly train whole slide images (WSI) on convolutional neural networks (CNNs). Most of the previous weakly supervision works divide high-resolution WSIs into small image patches and separately input them into the CNN to classify them as tumors or normal areas. The first difficulty is that although the method based on the CNN framework achieves a high accuracy rate, it increases the model parameters and computational complexity. The second difficulty is balancing the relationship between accuracy and model compu-tation. It makes the model maintain and improve the classification accuracy as much as possible based on the lightweight. In this paper, we propose a new lightweight architecture called Pyramid Tokens-to-Token VIsion Transformer (PyT2T-ViT) with multiple instance learning based on Vision Transformer. We introduce the feature extractor of the model with Token-to-Token ViT (T2T-ViT) to reduce the model parameters. The performance of the model is improved by combining the image pyramid of multiple receptive fields so that it can take into account the local and global features of the cell structure at a single scale. We applied the method to our collection of 560 thyroid pathology images from the same institution, model parameters and computation were greatly reduced. The classification effect is significantly better than the CNN-based method.