ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

Wang, Qilong; Wu, Banggu; Zhu, Pengfei; Li, Peihua; Zuo, Wangmeng; Hu, Qinghua

doi:10.48550/arxiv.1910.03151

Cited by 95 publications

(129 citation statements)

References 25 publications

Supporting

Mentioning

129

Contrasting

Order By: Relevance

“…In 2019, Hu et al [149] proposed the Efficient Channel Attention (ECA) module to solve the irrationality of channel dimensionality reduction from SE block [27] (Figure 23 F e x). Ref.…”

Section: Eca-netmentioning

confidence: 99%

“…Ref. [149] show avoiding dimensionality reduction is important for learning channel attention, and appropriate cross-channel interaction can preserve performance while significantly decreasing model complexity. ECA-Net proposes a local cross-channel interaction strategy without dimensionality reduction by using one-dimensional (1D) convolution, as shown in Figure 28.…”

Section: Eca-netmentioning

confidence: 99%

“…SENet [27] is one of the most successful examples of using the attention mechanism for image classification. Not only the above BAM [138], CBAM [139], GENet [140], SKNet [141], ECA-Net [149] and CA [150], but also GALA [151], AA-Net [152] and TA [153] developed its ideas by adopting different spatial attention mechanisms or designing advanced attention blocks. In addition, GSoP-Net [142], A 2 Net [148], NLNet [154], GCNet [155], SCNet [156] and CCNet [157] serve as typical examples of non-local/self-attention networks are recently very popular due to their capability of building spatial or channel-wise attention.…”

Section: Other Attention Modules and Summarymentioning

confidence: 99%

See 2 more Smart Citations

Review of Image Classification Algorithms Based on Convolutional Neural Networks

et al. 2021

View full text Add to dashboard Cite

Image classification has always been a hot research direction in the world, and the emergence of deep learning has promoted the development of this field. Convolutional neural networks (CNNs) have gradually become the mainstream algorithm for image classification since 2012, and the CNN architecture applied to other visual recognition tasks (such as object detection, object localization, and semantic segmentation) is generally derived from the network architecture in image classification. In the wake of these successes, CNN-based methods have emerged in remote sensing image scene classification and achieved advanced classification accuracy. In this review, which focuses on the application of CNNs to image classification tasks, we cover their development, from their predecessors up to recent state-of-the-art (SOAT) network architectures. Along the way, we analyze (1) the basic structure of artificial neural networks (ANNs) and the basic network layers of CNNs, (2) the classic predecessor network models, (3) the recent SOAT network algorithms, (4) comprehensive comparison of various image classification methods mentioned in this article. Finally, we have also summarized the main analysis and discussion in this article, as well as introduce some of the current trends.

show abstract

“…In 2019, Hu et al [149] proposed the Efficient Channel Attention (ECA) module to solve the irrationality of channel dimensionality reduction from SE block [27] (Figure 23 F e x). Ref.…”

Section: Eca-netmentioning

confidence: 99%

Section: Eca-netmentioning

confidence: 99%

Section: Other Attention Modules and Summarymentioning

confidence: 99%

See 1 more Smart Citation

Review of Image Classification Algorithms Based on Convolutional Neural Networks

et al. 2021

View full text Add to dashboard Cite

show abstract

“…We use the term adaptive feature extraction because in those methods, the main feature extraction process is supported by additional parametric or non-parametric functions. These functions are computed on the inputs to the network [8] or the inputs to each layer [11,12,13,23,26,34,32,35]. This allows those networks to be flexible to the context of the input, making the network more dynamic during inference.…”

Section: Related Workmentioning

confidence: 99%

End-To-End Data-Dependent Routing in Multi-Path Neural Networks

Tissera¹,

Wijessinghe²,

Vithanage³

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural networks are known to give better performance with increased depth due to their ability to learn more abstract features. Although the deepening of networks has been well established, there is still room for efficient feature extraction within a layer which would reduce the need for mere parameter increment. The conventional widening of networks by having more filters in each layer introduces a quadratic increment of parameters. Having multiple parallel convolutional/dense operations in each layer solves this problem, but without any context-dependent allocation of resources among these operations: the parallel computations tend to learn similar features making the widening process less effective. Therefore, we propose the use of multi-path neural networks with data-dependent resource allocation among parallel computations within layers, which also lets an input to be routed end-to-end through these parallel paths. To do this, we first introduce a cross-prediction based algorithm between parallel tensors of subsequent layers. Second, we further reduce the routing overhead by introducing feature-dependent cross-connections between parallel tensors of successive layers. Our multi-path networks show superior performance to existing widening and adaptive feature extraction, and even ensembles, and deeper networks at similar complexity in the image recognition task.

show abstract

“…Augmenting convolution models with self-attention Parmar et al, 2019;Wang et al, 2019) provides the model with the ability to capture global contexts in an image and has yielded gains in several vision tasks such as image classification and objective detection. We follow the protocols in , i.e.…”

Section: Image Classificationmentioning

confidence: 99%

SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection

Li,

Meng,

Zhou

et al. 2020

Preprint

View full text Add to dashboard Cite

While the self-attention mechanism has been widely used in a wide variety of tasks, it has the unfortunate property of a quadratic cost with respect to the input length, which makes it difficult to deal with long inputs. In this paper, we present a method for accelerating and structuring self-attentions: Sparse Adaptive Connection (SAC).In SAC, we regard the input sequence as a graph and attention operations are performed between linked nodes. In contrast with previous self-attention models with pre-defined structures (edges), the model learns to construct attention edges to improve task-specific performances. In this way, the model is able to select the most salient nodes and reduce the quadratic complexity regardless of the sequence length. Based on SAC, we show that previous variants of self-attention models are its special cases. Through extensive experiments on neural machine translation, language modeling, graph representation learning and image classification, we demonstrate SAC is competitive with state-of-the-art models while significantly reducing memory cost. 1

show abstract

ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

Cited by 95 publications

References 25 publications

Review of Image Classification Algorithms Based on Convolutional Neural Networks

Review of Image Classification Algorithms Based on Convolutional Neural Networks

End-To-End Data-Dependent Routing in Multi-Path Neural Networks

SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection

Contact Info

Product

Resources

About