MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features

Wadekar, Shakti N.; Chaurasia, Abhishek

doi:10.48550/arxiv.2209.15159

Cited by 14 publications

(19 citation statements)

References 26 publications

(47 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Following this, we bring the channel dimension back to its original size using a 1 × 1 convolutional layer and then concatenate it with the output feature of the local representation block. This strategy, distinct from MobileViT's concatenation of input and global representation features, has proven effective in MobileViTv3 (Wadekar and Chaurasia 2022). This effectiveness is attributed to the stronger correlation between local and global representation features, and the fact that local representation features possess a higher channel count compared to input features, allowing more channel information to be integrated into the fusion block.…”

Section: Swin Mobilevit Blockmentioning

confidence: 99%

Celiac disease diagnosis from endoscopic images based on multi-scale adaptive hybrid architecture model

Wang,

Shi,

Gao

et al. 2024

Phys. Med. Biol.

View full text Add to dashboard Cite

Background:Celiac disease has emerged as a significant global public health concern, exhibiting an estimated worldwide prevalence of approximately 1%. However, existing research pertaining to domestic occurrences of celiac disease is confined mainly to case reports and limited case analyses. Furthermore, there is a substantial population of undiagnosed patients in the Xinjiang region.Purpose:This study endeavors to develop a novel and focused deep learning model. The primary objective is to facilitate celiac disease diagnosis through the utilization of a dataset comprising endoscopic images obtained from patients with celiac disease in Xinjiang.Methods:We propose a novel CNN-Transformer hybrid architecture for deep learning, tailored to the diagnosis of celiac disease using endoscopic images. Within this architecture, a multi-scale spatial adaptive selective kernel convolution feature attention module demonstrates remarkable efficacy in diagnosing celiac disease. We dynamically capture salient features within the local channel feature map that correspond to distinct manifestations of endoscopic image lesions in the celiac disease-affected areas such as the duodenal bulb, duodenal descending segment, and terminal ileum. This process serves to extract and fortify the spatial information specific to different lesions. This strategic approach facilitates not only the extraction of diverse lesion characteristics but also the attentive consideration of their spatial distribution. Additionally, we integrate the global representation of the feature map obtained from the Transformer with the locally extracted information via convolutional layers. This integration achieves a harmonious synergy that optimizes the diagnostic prowess of the model.Results:Overall, the accuracy, specificity, F1-Score, and precision in the experimental results were 98.38%, 99.04%, 98.66% and 99.38%, respectively.Conclusion:This study introduces a deep learning network equipped with both global feature response and local feature extraction capabilities. This innovative architecture holds significant promise for the accurate diagnosis of celiac disease by leveraging endoscopic images captured from diverse anatomical sites.

show abstract

Section: Swin Mobilevit Blockmentioning

confidence: 99%

Celiac disease diagnosis from endoscopic images based on multi-scale adaptive hybrid architecture model

Wang,

Shi,

Gao

et al. 2024

Phys. Med. Biol.

View full text Add to dashboard Cite

show abstract

“…The YOLOv8s architecture, a well-established detector, consists of three integral components: a backbone network, neck, and head. To optimally configure this detector for infrared imagery, we engineered an augmented version, which entails incorporating a specialized infrared feature extraction module and supplanting the standard backbone with the advanced MobileVITv3 [31] network. This dedicated infrared feature extraction module is adept at discerning salient features unique to infrared imagery, thereby enriching the feature representation.…”

Section: Detectorsmentioning

confidence: 99%

“…As shown in Figure 4, MobileViTv3 [31] is the third generation model of MobileViT, a lightweight vision model for mobile, which combines CNNs and visual transformers (ViTs) to outperform the accuracy of a model with around 8 million covariates on the classification task with around 6 million covariates through a basic training methodology, and outperform the vast majority of mainstream models. Compared to the Darknet-53, the backbone network of YOLOv8s, MobileViTv3 features a more lightweight structure that can achieve better performance while reducing both the model size and computational demands.…”

Section: Mobilevitv3 Backbone Networkmentioning

confidence: 99%

Infrared multi‐target detection and tracking in dense urban traffic scenes

Zha,

Luo,

2024

IET Image Processing

View full text Add to dashboard Cite

Infrared object detection and tracking in dense urban traffic remain a challenge due to factors such as low contrast, small intra‐class differences, and frequent false positives and negatives. To overcome these, the authors introduce YOLO‐IR, an algorithm based on the enhanced YOLOv8s, and YOLO‐DeepOC‐IR, a comprehensive infrared multi‐object tracking method for urban traffic, integrating both detection and tracking. During preprocessing, three infrared image enhancement techniques, local contrast multi‐scale enhancement, non‐local means, and contrast limited adaptive histogram equalization, are applied for better reliability in dense scenes. To further improve the performance, the original YOLOv8s backbone is replaced with MobileVITv3 to enhance detection accuracy and robustness. This infrared feature extraction module, incorporated into the detector, combines canny edge detection, Gabor filtering, and open operation layers, significantly boosting object detection in infrared imagery. The tracker's feature processing capabilities are improved using the learned arrangements of three patch codes descriptor and locality‐sensitive hashing for feature extraction and matching. Experimental results on FLIR ADAS v2 and InfiRay datasets indicate superior performance of this method, achieving 78.6% mAP and 151.1 FPS in detection, and up to 80.8% moving object tracking accuracy, 78.6% identification F1 score, and 62.1% higher order tracking accuracy in multi‐object tracking.

show abstract

“…Subsequently, MobileVitv2 [39] proposed a linear time complexity SA method, which greatly reduced the resource occupation of the model. MobileVitv3 [40] improved the generalisation of the model by integrating context multi-scale features. Hybrid model makes full use of CNN's global field of vision of local bias and Transformer structure.…”

Section: Efficient Modelmentioning

confidence: 99%

Printed circuit board solder joint quality inspection based on lightweight classification network

Zhang,

Zhu

et al. 2023

IET Cyber-Syst and Robotics

View full text Add to dashboard Cite

Solder joint quality inspection is a crucial step in the qualification inspection of printed circuit board (PCB) components, and efficient and accurate inspection methods will greatly improve its production efficiency. In this paper, we propose a PCB solder joint quality detection algorithm based on a lightweight classification network. First, the Select Joint segmentation method was used to obtain the solder joint information, and colour space conversion was used to locate the solder joint. The mask method, contour detection, and box line method were combined to complete the extraction of solder joint information. Then, by combining the respective characteristics of convolutional neural network and Transformer and introducing Cross‐covariance attention to reduce the computational complexity and resource consumption of the model and evenly distribute the global view mutual information in the whole training process, a new lightweight network model MobileXT is proposed to complete defect classification. Only 16.4% of the Vision Transformer computing resources used in this model can achieve an average accuracy improvement of 31%. Additionally, the network is trained and validated using a dataset of 1804 solder joint images constructed from 93 PCB images and two external datasets to evaluate MobileXT performance. The proposed method achieves more efficient localization of the solder joint information and more accurate classification of weld joint defects, and the lightweight model design is more appropriate for industrial edge device deployments.

show abstract

MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features

Cited by 14 publications

References 26 publications

Celiac disease diagnosis from endoscopic images based on multi-scale adaptive hybrid architecture model

Celiac disease diagnosis from endoscopic images based on multi-scale adaptive hybrid architecture model

Infrared multi‐target detection and tracking in dense urban traffic scenes

Printed circuit board solder joint quality inspection based on lightweight classification network

Contact Info

Product

Resources

About