Multi-Dimension Compression of Feed-Forward Network in Vision Transformers

Shipeng Bai, Jun Chen, Yu Yang, Yong Liu

Abstract: Highlights•FFN Irregular Sparsity: A new sparse method that replaces the insignificant elements with zero.•Convert to Structure Sparsity: It converts irregular sparsity into regular sparsity.•X-Transformation: A unified converting method further accelerates the model.•Extensive experiments on ImageNet with Swin-Transformer and DeiT demonstrate the effectiveness of our method.