AttendNets: Tiny Deep Image Recognition Neural Networks for the Edge via Visual Attention Condensers

Abstract

While significant advances in deep learning has resulted in state-of-the-art performance across a large number of complex visual perception tasks, the widespread deployment of deep neural networks for TinyML applications involving on-device, low-power image recognition remains a big challenge given the complexity of deep neural networks. In this study, we introduce AttendNets, low-precision, highly compact deep neural networks tailored for on-device image recognition. More specifically, AttendNets possess deep self-attention architectures based on visual attention condensers, which extends on the recently introduced stand-alone attention condensers to improve spatial-channel selective attention. Furthermore, At-tendNets have unique machine-designed macroarchitecture and microarchitecture designs achieved via a machine-driven design exploration strategy. Experimental results on ImageNet 50 benchmark dataset for the task of on-device image recognition showed that AttendNets have significantly lower architectural and computational complexity when compared to several deep neural networks in research literature designed for efficiency while achieving highest accuracies (with the smallest AttendNet achieving ∼7.2% higher accuracy, while requiring ∼3× fewer multiplyadd operations, ∼4.17× fewer parameters, and ∼16.7× lower weight memory requirements than MobileNet-V1). Based on these promising results, AttendNets illustrate the effectiveness of visual attention condensers as building blocks for enabling various on-device visual perception tasks for TinyML applications.

TB-Net: A Tailored, Self-Attention Deep Convolutional Neural Network Design for Detection of Tuberculosis Cases From Chest X-Ray Images

“…Third, it can be observed that a majority of the layers of the TB-Net network architecture design is comprised of visual attention condensers (Wong et al, 2020b ), which are a variant of the highly efficient attention condenser self-attention mechanisms recently introduced in Wong et al ( 2020a ). More specifically, visual attention condensers produce condensed embedding characterizing joint spatial and cross-channel activation relationships and achieves selective attention accordingly to improve representational capability while maintaining very low architectural and computational complexity.…”

CellDefectNet: A Machine-designed Attention Condenser Network for Electroluminescence-based Photovoltaic Cell Defect Inspection

“…We take into account several computational and bestpractice constraints which are formulated via the indicator function 1 g (•): i) the macroarchitecture design uses several parallel columns to significantly reduce the architectural and computational complexity with much greater disentanglement of learned features; ii) to reduce the considerable information loss caused by the pointwise strided convolutions used in residual networks [6] and RegNet architecture [10] here we restricted its use from the optimization; iii) antialiasing downsampling (AADS) [24] modules are to be used in the early network stage to improve network stability and robustness; iv) FLOPs within 20% of 100M FLOPs for edge compute scenarios. In the machine driven design exploration process, attention condensers (VAC) [20], [21] are used as an highly efficient self-attention module to learn and produce condensed embedding characterizing the joint local and cross-channel activation relationships. However, the machine-driven design exploration process automatically determines the best way to satisfy the defined constraints in designing the CellDefectNet architecture.…”

“…Early-stage self-attention: the VACs are leveraged heavily within the initial modules used in the network architecture. VAC was first introduced by Wong et al[21] for image classification. The VACs can help to better model activation relationships and improves selective attention.…”

Cancer-Net SCa: tailored deep neural network designs for detection of skin cancer from dermoscopy images

“…The number in each convolution module represents the number of channels. The numbers in each visual attention condenser represents the number of channels for the down-mixing layer, the embedding structure, and the up-mixing layer, respectively (details can be found in [15]). It can be observed that all Cancer-Net SCa architectures exhibit both great macroarchitecture and microarchitecture design diversity, with certain models exhibiting specific lightweight macroarchitecture design characteristics such as attention condenser and projection-expansion-projection-expansion (PEPE) design patterns comprised of channel dimensionality reduction, depthwise convolutions, and pointwise convolutions are highly diverse and heterogeneous, with a mix of spatial convolutions, pointwise convolutions, and depthwise convolutions, all with different microarchitecture designs.…”

“…The use of computer vision and machine learning for the diagnosis of pigmented skin lesions has been shown to be accurate and practical [5][6][7][9][10][11][12][13], and can improve biopsy decision making [10], as well as act as a pre-screening tool to reduce the amount of a time a professional spends on each case. Motivated by the challenge of skin cancer detection, and inspired by the open source and open access efforts of the research community, in this study we introduce Cancer-Net SCa, a suite of deep neural network designs tailored for the detection of skin cancer from dermoscopy images, one of which possesses a self-attention architecture design with attention condensers [14,15]. To construct Cancer-Net SCa, we leveraged a machine-driven design strategy that leverages human experience and ingenuity with the meticulousness and raw speed of machines.…”