Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet

Wieland Brendel, Matthias Bethge

Abstract: Deep Neural Networks (DNNs) excel on many complex perceptual tasks but it has proven notoriously difficult to understand how they reach their decisions. We here introduce a high-performance DNN architecture on ImageNet whose decisions are considerably easier to explain. Our model, a simple variant of the ResNet-50 architecture called BagNet, classifies an image based on the occurrences of small local image features without taking into account their spatial ordering. This strategy is closely related to the bag-of-feature (BoF) models popular before the onset of deep learning and reaches a surprisingly high accuracy on ImageNet (87.6% top-5 for 32 x 32 px features and Alexnet performance for 16 x16 px features). The constraint on local features makes it straight-forward to analyse how exactly each part of the image influences the classification. Furthermore, the BagNets behave similar to state-of-the art deep neural networks such as VGG-16, ResNet-152 or DenseNet-169 in terms of feature sensitivity, error distribution and interactions between image parts. This suggests that the improvements of DNNs over previous bag-of-feature classifiers in the last few years is mostly achieved by better fine-tuning rather than by qualitatively different decision strategies.

Keywords: interpretability, representation learning, bag of features, deep learning, object recognition

TL;DR: Aggregating class evidence from many small image patches suffices to solve ImageNet, yields more interpretable models and can explain aspects of the decision-making of popular DNNs.

Code: [ wielandbrendel/bag-of-local-features-models](https://github.com/wielandbrendel/bag-of-local-features-models) + [ 3 community implementations](https://paperswithcode.com/paper/?openreview=SkfMWhAqYQ)

Community Implementations: [ 5 code implementations](https://www.catalyzex.com/paper/approximating-cnns-with-bag-of-local-features/code)