[UniProt Challenge] CNN + Attention

Attention-augmented CNN

[1904.09925] Attention Augmented Convolutional Networks (arxiv.org)

Attention Augmented Convolutional Networks

 

 

Convolutional networks have been the paradigm of choice in many computer vision applications. The convolution operation however has a significant weakness in that it only operates on a local neighborhood, thus missing global information. Self-attention, on the other hand, has emerged as a recent advance to capture long range interactions, but has mostly been applied to sequence modeling and generative modeling tasks. In this paper, we consider the use of self-attention for discriminative visual tasks as an alternative to convolutions. We introduce a novel two-dimensional relative self-attention mechanism that proves competitive in replacing convolutions as a stand-alone computational primitive for image classification. We find in control experiments that the best results are obtained when combining both convolutions and self-attention. We therefore propose to augment convolutional operators with this self-attention mechanism by concatenating convolutional feature maps with a set of feature maps produced via self-attention. Extensive experiments show that Attention Augmentation leads to consistent improvements in image classification on ImageNet and object detection on COCO across many different models and scales, including ResNets and a state-of-the art mobile constrained network, while keeping the number of parameters similar. In particular, our method achieves a 1.3% top-1 accuracy improvement on ImageNet classification over a ResNet50 baseline and outperforms other attention mechanisms for images such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 mAP in COCO Object Detection on top of a RetinaNet baseline.

 

Vision Transformer

[2010.11929] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (arxiv.org)

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

 

SE-NET

 

InChI / Resnet + LSTM with attention / starter | Kaggle

Attention code 참조

 

 

 

Reference

1. Transformers in Vision： A Survey [1] Transformer 소개 & Transformers for Image Recognition (hoya012.github.io)

Transformers in Vision： A Survey [1] Transformer 소개 & Transformers for Image Recognition