Correlation Net: Spatiotemporal multimodal deep learning for action recognition

Abstract

This letter describes a network that is able to capture multimodal correlations over arbitrary timestamps. The proposed scheme operates as a complementary, extended network over multimodal CNN. For action recognition, the spatial and temporal streams are vital components of deep Convolutional Neural Network (CNNs), but reducing the occurrence of overfitting and fusing these two streams remain open problems. The existing fusion approach is to average the two streams. To this end, we propose a correlation network with a Shannon fusion to learn a CNN that has already been trained. Long-range video may consist of spatiotemporal correlation over arbitrary times. This correlation can be captured using simple fully connected layers to form the correlation network. This is found to be complementary to the existing network fusion methods. We evaluate our approach on the UCF-101 and HMDB-51 datasets, and the resulting improvement in accuracy demonstrates the importance of multimodal correlation.

Multi-head attention-based two-stream EfficientNet for action recognition

“…The main building block of EfficientNet-B0 is the mobile inverted bottleneck (MBConv), which is based on the concept of MobileNet [54,55]. As shown in Fig.…”

“…As shown in Fig. 3, MBConv consists of two convolutional layers(k1 × 1), a depthwise convolutional layer, a Squeeze and Excitation (SE) [54,55] block, and a dropout layer. The first convolutional layer is used to expand the channels.…”

“…This indicates that the multi-head attention mechanism is useful for recognizing the action, and the proposed MAT-EffNet is a competitive network for action recognition. [12] 63.3 -I3D-RGB [12] 72.1 90.3% ARTNet [52] 70.7 89.3% MoViNet-A5 [66] 71.7 -VidTr-L [67] 70.2 89% MAT-EffNet 72.6 90.8% LRCN [38] RGB + optical flow 82.9 -C3D [26] RGB only + 3D CNNs 85.2 -IDTs [32] RGB only + 3D CNNs 85.9 57.2% Two-stream [1] RGB + optical flow 88.0 59.4% FSTCN [39] RGB + optical flow 88.1 59.1% P3D-199 [65] RGB + 3D CNNs 89.2 62.9% TDD [34] RGB + optical flow 90.3 63.2% STS-network [17] RGB + optical flow + others 90.1 62.4% R-M3D [11] RGB only + 3D CNNs 93.2 65.4% STDAN + RGB difference [58] RGB + optical flow + others 91.0 60.4% TSN Corrnet [55] RGB + optical flow 94.4 70.6% MSM-ResNets [56] RGB + optical flow + others 93.5 66.7% R-STAN-50 [68] RGB + optical flow 91.…”

“…As shown in Table 6, we compare MAT-EffNet with several reference approaches on the UCF101 and HMDB51 datasets. We compare our approach with both conventional approaches and deep learning-based approaches such as long-term recurrent convolutional networks (LRCN) [37], 3D convolutional networks (C3D) [25], improved trajectories (iDTs) [31], two-stream neural network (Two-stream) [1], factorized spatio-temporal convolutional network (FSTCN) [38], trajectory-pooled deep-convolutional descriptors (TDD) [33], spatiotemporal saliency-based multi-stream network (STSnetwork) [24], multi-cue-based 3D residual network (R-M3D) [11], motion saliency-based multi-stream multiplier ResNets (MSM-ResNets) [56] and correlational convolutional LSTM network (TSN Corrnet) [55].…”

Improved Soccer Action Spotting using both Audio and Video Streams

“…Recently, research in multimodal models use, in addition to the RGB video streams, information about the motion within the video sequences: the optical flow can be used [77,79] or even player pose sequences [8,71]. For golf and tennis tournaments, a multimodal architecture using the reactions (such as high fives or fist pumps) and expressions of the players (aggressive, smiling, etc.…”