Tree-structured Attention Module for Image Classification

Gyungin Shin, Sung-Ho Bae, and Yong-Jae Moon

Keywords: inter-channel relationship, attention module, point-wise group convolution

TL;DR: Our paper proposes an attention module which captures inter-channel relationships and offers large performance gains.

Abstract: Recent studies in attention modules have enabled higher performance in computer vision tasks by capturing global contexts and accordingly attending important features. In this paper, we propose a simple and highly parametrically efficient module named Tree-structured Attention Module (TAM) which recursively encourages neighboring channels to collaborate in order to produce a spatial attention map as an output. Unlike other attention modules which try to capture long-range dependencies at each channel, our module focuses on imposing non-linearities be- tween channels by utilizing point-wise group convolution. This module not only strengthens representational power of a model but also acts as a gate which controls signal flow. Our module allows a model to achieve higher performance in a highly parameter-efficient manner. We empirically validate the effectiveness of our module with extensive experiments on CIFAR-10/100 and SVHN datasets. With our proposed attention module employed, ResNet50 and ResNet101 models gain 2.3% and 1.2% accuracy improvement with less than 1.5% parameter over- head. Our PyTorch implementation code is publicly available.

Code: https://github.com/NoelShin/TAM