ConViT: improving vision transformers with soft convolutional inductive biases*

d’Ascoli, Stéphane; Touvron, Hugo; Leavitt, Matthew L.; Morcos, Ari S.; Biroli, Giulio; Sagun, Levent

doi:10.1088/1742-5468/ac9830

Cited by 289 publications

(131 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Models. We compared humans with 84 different DNNs representing the variety of approaches used in the field today: 50 CNNs trained on ImageNet [1,49,[53][54][55][56][57][58][59][60][61][62][63][64][64][65][66][67][68][69][70][71][72][73], 6 CNNs trained on other datasets in addition to ImageNet (which we refer to as "CNN extra data") [1,66,74], 10 vision transformers [75][76][77][78][79], 6 CNNs trained with self-supervision [80,81], and 13 models trained for robustness to noise or adversarial examples [82,83]. We used pretrained weights for each of these models supplied by their authors, with a variety of licenses (detailed in SI §2), implemented in Tensorflow 2.0, Keras, or PyTorch.…”

Section: Methodsmentioning

confidence: 99%

Harmonizing the object recognition strategies of deep neural networks with humans

Fel¹,

Felipe²,

Linsley³

et al. 2022

Preprint

View full text Add to dashboard Cite

The many successes of deep neural networks (DNNs) over the past decade have largely been driven by computational scale rather than insights from biological intelligence. Here, we explore if these trends have also carried concomitant improvements in explaining the visual strategies humans rely on for object recognition. We do this by comparing two related but distinct properties of visual strategies in humans and DNNs: where they believe important visual features are in images and how they use those features to categorize objects. Across 84 different DNNs trained on ImageNet and three independent datasets measuring the where and the how of human visual strategies for object recognition on those images, we find a systematic trade-off between DNN categorization accuracy and alignment with human visual strategies for object recognition. State-of-the-art DNNs are progressively becoming less aligned with humans as their accuracy improves. We rectify this growing issue with our neural harmonizer: a general-purpose training routine that both aligns DNN and human visual strategies and improves categorization accuracy. Our work represents the first demonstration that the scaling laws [1-3] that are guiding the design of DNNs today have also produced worse models of human vision. We release our code and data at https://serre-lab.github.io/Harmonization to help the field build more human-like DNNs.

show abstract

Section: Methodsmentioning

confidence: 99%

Harmonizing the object recognition strategies of deep neural networks with humans

Fel¹,

Felipe²,

Linsley³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For the leaning layers, the gating parameter eventually converges to close to 0, indicating that the convolutional induction bias is effectively ignored. However, for the starting layers, many attention heads maintain high gating values, suggesting that the network uses the convolutional induction bias of the earlier layers to aid training [ 43 ].…”

Section: Methodsmentioning

confidence: 99%

Landslide Susceptibility Mapping by Fusing Convolutional Neural Networks and Vision Transformer

Bao

Liu

Wang

et al. 2022

Sensors

View full text Add to dashboard Cite

Landslide susceptibility mapping (LSM) is an important decision basis for regional landslide hazard risk management, territorial spatial planning and landslide decision making. The current convolutional neural network (CNN)-based landslide susceptibility mapping models do not adequately take into account the spatial nature of texture features, and vision transformer (ViT)-based LSM models have high requirements for the amount of training data. In this study, we overcome the shortcomings of CNN and ViT by fusing these two deep learning models (bottleneck transformer network (BoTNet) and convolutional vision transformer network (ConViT)), and the fused model was used to predict the probability of landslide occurrence. First, we integrated historical landslide data and landslide evaluation factors and analysed whether there was covariance in the landslide evaluation factors. Then, the testing accuracy and generalisation ability of the CNN, ViT, BoTNet and ConViT models were compared and analysed. Finally, four landslide susceptibility mapping models were used to predict the probability of landslide occurrence in Pingwu County, Sichuan Province, China. Among them, BoTNet and ConViT had the highest accuracy, both at 87.78%, an improvement of 1.11% compared to a single model, while ConViT had the highest F1-socre at 87.64%, an improvement of 1.28% compared to a single model. The results indicate that the fusion model of CNN and ViT has better LSM performance than the single model. Meanwhile, the evaluation results of this study can be used as one of the basic tools for landslide hazard risk quantification and disaster prevention in Pingwu County.

show abstract

“…Thus, some combinations of transformers and CNNs have been proposed for reducing the computational cost and number of parameters [ 29 ]. D’Ascoli et al introduced a positional self-attention mechanism equipped with convolutional inductive bias, adjusting the attention to positional and contextual information through learnable gating parameters [ 30 ]. BotNet [ 31 ] embedded multi-head self-attention (MHSA) in three bottlenecks of C5 in ResNet-50 [ 32 ] and outstood in object detection and instance segmentation tasks [ 33 ].…”

Section: Related Workmentioning

confidence: 99%

YOLOv5s-FP: A Novel Method for In-Field Pear Detection Using a Transformer Encoder and Multi-Scale Collaboration Perception

Rao

Jin

et al. 2022

Sensors

View full text Add to dashboard Cite

Precise pear detection and recognition is an essential step toward modernizing orchard management. However, due to the ubiquitous occlusion in orchards and various locations of image acquisition, the pears in the acquired images may be quite small and occluded, causing high false detection and object loss rate. In this paper, a multi-scale collaborative perception network YOLOv5s-FP (Fusion and Perception) was proposed for pear detection, which coupled local and global features. Specifically, a pear dataset with a high proportion of small and occluded pears was proposed, comprising 3680 images acquired with cameras mounted on a ground tripod and a UAV platform. The cross-stage partial (CSP) module was optimized to extract global features through a transformer encoder, which was then fused with local features by an attentional feature fusion mechanism. Subsequently, a modified path aggregation network oriented to collaboration perception of multi-scale features was proposed by incorporating a transformer encoder, the optimized CSP, and new skip connections. The quantitative results of utilizing the YOLOv5s-FP for pear detection were compared with other typical object detection networks of the YOLO series, recording the highest average precision of 96.12% with less detection time and computational cost. In qualitative experiments, the proposed network achieved superior visual performance with stronger robustness to the changes in occlusion and illumination conditions, particularly providing the ability to detect pears with different sizes in highly dense, overlapping environments and non-normal illumination areas. Therefore, the proposed YOLOv5s-FP network was practicable for detecting in-field pears in a real-time and accurate way, which could be an advantageous component of the technology for monitoring pear growth status and implementing automated harvesting in unmanned orchards.

show abstract

ConViT: improving vision transformers with soft convolutional inductive biases*

Cited by 289 publications

References 42 publications

Harmonizing the object recognition strategies of deep neural networks with humans

Harmonizing the object recognition strategies of deep neural networks with humans

Landslide Susceptibility Mapping by Fusing Convolutional Neural Networks and Vision Transformer

YOLOv5s-FP: A Novel Method for In-Field Pear Detection Using a Transformer Encoder and Multi-Scale Collaboration Perception

Contact Info

Product

Resources

About