Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks

Guo, Minghao; Liu, Zheng-Ning; Mu, Tai‐Jiang; Hu, Shi‐Min

doi:10.1109/tpami.2022.3211006

Cited by 233 publications

(92 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the course of continuous learning, we have learned that the deep learning is regarded as a black box, which may not be explained in biological processes [44]. In the following work, we will take biological interpretation into consideration and apply more effective attention modules including the convolutional block attention module (CBAM) [45], external attention (EA) [46], and so on, through which will have more meaningful gains in the following experiments.…”

Section: Discussionmentioning

confidence: 99%

pSuc-EDBAM: Predicting lysine succinylation sites in proteins based on ensemble dense blocks and an attention module

Qiu

2022

Preprint

View full text Add to dashboard Cite

Background Lysine succinylation is a newly discovered protein post-translational modifications. Predicting succinylation sites helps investigate the metabolic disease treatments. However, the biological experimental approaches are costly and inefficient, it is necessary to develop efficient computational approaches. Results In this paper, we proposed a novel predictor based on ensemble dense blocks and an attention module, called as pSuc-EDBAM, which adopted one hot encoding to derive the feature maps of protein sequences. Afterward, the ensemble dense blocks were used to capture feature information at different levels in the process of feature learning. We also introduced an attention module to evaluate the importance degrees of different features. The experimental results show that Acc reaches 74.25%, and MCC reaches 0.2927 on the testing dataset, which suggest that the pSuc-EDBAM outperforms the existing predictors. To make the convenience of the experimental scientists, a user-friendly web server has been established (http://bioinfo.wugenqiang.top/pSuc-EDBAM/), by which the desired results can be easily obtained. Conclusions The experimental results of 10-fold cross-validation on the training dataset and independent test on the testing dataset showed that pSuc-EDBAM outperforms the existing succinylation site predictors and can predict potential succinylation sites effectively. The pSuc-EDBAM is feasible and obtains the credible predictive results, which may also provide valuable references for other related research.

show abstract

Section: Discussionmentioning

confidence: 99%

pSuc-EDBAM: Predicting lysine succinylation sites in proteins based on ensemble dense blocks and an attention module

Qiu

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This hybriding approach can be further enhanced by attention algorithms in terms of the feature extraction process. A fused attention mechanism that consisted of self-attention (SA) [113] and external-attention (EA) [114] is used in [115]. The SA was used to weigh the features based on their importance, while EA was used to discover the correlation between different features.…”

Section: Based Hybrid Modelsmentioning

confidence: 99%

Deep learning in fault detection and diagnosis of building HVAC systems: A systematic review with meta analysis

Zhang

Saeed²,

Sadeghian³

2023

Energy and AI

View full text Add to dashboard Cite

“…In this subsection, we introduce our proposed EACNN network. Unlike the traditional CNN2, whose convolution layer learns features only in a fixed receptive field, the key of EACNN is to employ the External Attention [8] module to learn features with a learnable receptive field and capture long-term dependencies, which can detect the correlation in whole signals.…”

Section: Eacnn Networkmentioning

confidence: 99%

“…In [7], the attention mechanism is utilized to perform the recalibration. In [8], the authors proposed an external attention framework for image classification and object detection, which has better performance and lower computational cost than the self-attention mechanism. In addition, compared with CNN, the attention mechanism can pay more attention to global correlations.…”

mentioning

confidence: 99%

External Attention Mechanism-Based Modulation Classification

Tao

Shao

et al. 2023

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

This paper considers the modulation classification of radio frequency (RF) signals. An external attention mechanism-based convolution neural network (EACNN) is proposed. Thanks to the external attention layers, the EACNN network can capture the potential correlations of different modulation data, which helps reduce computational consumption and memory costs efficiently during training. Moreover, to account for the variation of the signals induced by channel fading, we further propose a customized batch normalization (BN) layer in EACNN to improve the classification accuracy with less training time. Numerical experiments on RML2016.a dataset shows that the proposed method outperforms the baseline method CNN2 by 7% in terms of classification accuracy.

show abstract

Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks

Cited by 233 publications

References 78 publications

pSuc-EDBAM: Predicting lysine succinylation sites in proteins based on ensemble dense blocks and an attention module

pSuc-EDBAM: Predicting lysine succinylation sites in proteins based on ensemble dense blocks and an attention module

Deep learning in fault detection and diagnosis of building HVAC systems: A systematic review with meta analysis

External Attention Mechanism-Based Modulation Classification

Contact Info

Product

Resources

About