Convolutional Neural Networks or Vision Transformers: Who Will Win the Race for Action Recognitions in Visual Data?

Abstract

Understanding actions in videos remains a significant challenge in computer vision, which has been the subject of several pieces of research in the last decades. Convolutional neural networks (CNN) are a significant component of this topic and play a crucial role in the renown of Deep Learning. Inspired by the human vision system, CNN has been applied to visual data exploitation and has solved various challenges in various computer vision tasks and video/image analysis, including action recognition (AR). However, not long ago, along with the achievement of the transformer in natural language processing (NLP), it began to set new trends in vision tasks, which has created a discussion around whether the Vision Transformer models (ViT) will replace CNN in action recognition in video clips. This paper conducts this trending topic in detail, the study of CNN and Transformer for Action Recognition separately and a comparative study of the accuracy-complexity trade-off. Finally, based on the performance analysis’s outcome, the question of whether CNN or Vision Transformers will win the race will be discussed.

An instance-based deep transfer learning method for quality identification of Longjing tea from multiple geographical origins

“…By considering the global information of the image, ViT is more competitive in some visual classification tasks [44,45]. Scholars have proved that ViT can be used in traffic sign classification [46], plant disease detection [47], and face recognition [48]. Vit-based transfer learning begins to attract more and more attention [49][50][51].…”

The Key Criteria for Predicting Unusual Behavior in the Elderly With Deep Learning Models Under 5G Technology

“…It is a subset of neural networks typically employed in contexts of three dimensional units, namely the height, width and depth used for image analysis and also applies to object classification [43]. Since CNNs can learn directly from the raw time series data, extract features from sequences of observations, and require neither domain expertise nor manually engineered input characteristics this current study plan to utilise it to detect the attributes associated to the activities of elderly people in determining their unusual behavior [44].…”

Potential roles of transformers in brain tumor diagnosis and treatment

“…However, CNN has a few weaknesses, including a slowness that is brought on by the max pooling operation; additionally, in contrast to the Transformer, it does not consider several perspectives that can be gained by learning, [121] which leads to disregard for global knowledge. Because it offers solutions to CNN's numerous weaknesses, Transformer has quickly become CNN's most formidable opponent [122] . The capability of the Transformer to prioritize relevant content while minimizing the repetition of unimportant content is its strength [123] .…”

“…However, it should be mentioned that both algorithms (i.e., CNN and Transformer) have their own shortcomings and benefits and it is still difficult to determine who will win this race. Nevertheless, the hybrid method, which is the most attractive formula because it enables us to take advantage of a model's strengths while simultaneously reducing the effects of that model's downsides, is more efficient and cost‐effective [122] . It combines CNN with transformers to provide a reliable model.…”

“…Because it offers solutions to CNN's numerous weaknesses, Transformer has quickly become CNN's most formidable opponent. [122] The capability of the Transformer to prioritize relevant content while minimizing the repetition of unimportant content is its strength. [123] Furthermore, since less demand is placed on the processing power, the visual characteristics are dynamically re-weighted based on the context.…”