(A) study for efficient spatial attention module

Abstract

Spatial attention module is widely used in deep neural networks. The monumental attention module,Transformer [11] was proposed as self-attention and encoder-decoder frameworks in machine translationtask, and improving capability of learning long-range dependencies. After that, spatial attention modulessuch as Non-local block [12] and Criss-cross attention block [5] were also proposed and improving per-formance in vision fields including action recognition, segmentation, and object detection. Despite greatsuccess, spatial attention can only be used in a limited way due to expensive computation and memorycosts. To reduce overhead, segmentation or pooling should be used and several methods were proposed,but stacking enough blocks is still limited. Also, multi-head attention is not used in vision and videotasks due to the memory limitation. In this paper, we analyze which factors play an important role inlearning spatial attention module in the view of geometric definition, and propose ‘Transposed attentionmodule’ that is faster and smaller than Non-local block [12] in the same number of blocks and heads. Thebiggest advantage of our module is that memory and speed are maintained when the number of headsis increasing. Also, we introduce new approach to interpret spatial attention module and get superiorperformance compared to Non-local block [12] on CIFAR-10, CIFAR-100 [9], and Tiny-ImageNet.