SparseSwin: Swin transformer with sparse transformer block

Krisna Pinasthika, Blessius Sheldo Putra Laksono, Riyandi Banovbi Putera Irsal, Syifa Hukma Shabiyya, Novanto Yudistira

Abstract: Highlights•Proposing Sparse Transformer Block with limited latent tokens to enhance Swin Transformer efficiency and performance.•We examined the effect of regularization on the attention weight obtained in SparTa Block.•We obtained accuracy improvement on ImageNet100, CIFAR10, and CIFAR100 benchmark datasets.•We obtained a smaller parameter size compared to the state-of-the-art Transformer model.