Discrete Representations Strengthen Vision Transformer RobustnessDownload PDF

29 Sept 2021, 00:31 (edited 14 Mar 2022)ICLR 2022 PosterReaders: Everyone
  • Keywords: vision transformer, robustness, image recognition
  • Abstract: Vision Transformer (ViT) is emerging as the state-of-the-art architecture for image recognition. While recent studies suggest that ViTs are more robust than their convolutional counterparts, our experiments find that ViTs are overly reliant on local features (\eg, nuisances and texture) and fail to make adequate use of global context (\eg, shape and structure). As a result, ViTs fail to generalize to out-of-distribution, real-world data. To address this deficiency, we present a simple and effective architecture modification to ViT's input layer by adding discrete tokens produced by a vector-quantized encoder. Different from the standard continuous pixel tokens, discrete tokens are invariant under small perturbations and contain less information individually, which promote ViTs to learn global information that is invariant. Experimental results demonstrate that adding discrete representation on four architecture variants strengthens ViT robustness by up to 12\% across seven ImageNet robustness benchmarks while maintaining the performance on ImageNet.
  • One-sentence Summary: We present a simple and effective architecture modification to ViT's input layer with discrete token representations.
15 Replies

Loading