DEEPAM: Toward Deeper Attention Module in Residual Convolutional Neural Networks

Shanshan Zhong, Wushao Wen, Jinghui Qin, Zhongzhan Huang

Abstract: The efficacy of depth in boosting the performance of residual convolutional neural networks (CNNs) has been well-established through abundant empirical or theoretical evidences. However, despite the attention module (AM) being a crucial component for high-performance CNNs, most existing research primarily focuses on their structural design, overlooking a direct investigation into the impact of AM depth on performance. Therefore, in this paper, we explore the influence of AM depth under various settings in detail. We observe that (1) appropriately increasing AM depth significantly boosts performance; (2) deepening AM exhibits a higher cost-effectiveness compared to traditional backbone deepening. However, deepening AM introduces inherent challenges in terms of parameter and inference cost. To mitigate them while enjoying the benefit of deepening AM, we propose a novel AM called DEEPAM, leveraging mechanisms from recurrent neural networks and the design of lightweight AMs. Extensive experiments on widely-used benchmarks and popular attention networks validate the effectiveness of our proposed DEEPAM.