Resource Efficient 3D Convolutional Neural Networks

Köpüklü, Okan; Köse, Neslihan; Gunduz, Ahmet; Rigoll, Gerhard

doi:10.1109/iccvw.2019.00240

Cited by 164 publications

(64 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The following points are noted: Table 12. Comparisons with the-state-of-the art methods on the UCF101 and HMDB51 datasets over 3 splits Algorithm UCF101 HMDB51 C3D [34] 85.2% --Soft Attention + LSTM [13] --41.3% Two-Stream + LSTM [38] 88.6% --TDD+FV [19] 90.3% 63.2% RNN+FV [58] 88.0% 54.3% LTC [33] 91.7% 64.8% ST-ResNet [29] 93.5% 66.4% TSN (BN-Inception) [28] 94.0% 68.5% TSN (Journal version) [78] 94.9% 71.0% AdaScan [59] 89.4% 54.9% ActionVLAD [39] 92.7% 66.9% TLE (BN-Inception) [40] 95.6% 71.1% Attention Cluster (ResNet-152) [47] 94.6% 69.2% TesNet (ImageNet pre-trained) [84] 95.2% 71.5% Two-in-one two stream [85] 92.0% --R(2+1) [86] 85.8% 54.8% 3D-SqueezeNet [87] 74.9% --Algorithm in [88](Pre-trained on Kinetics) 61.2% 33.4% TSM [89] 95.9% 73.5% T-STFT [90] 94.7% 71.5% TEA [91] 96.9%  The results of our methods are much better than the result 13 of the attention-based method, soft attention + LSTM [13].  Our method with BN-Inception as the backbone outperforms the conference version of TSN [28] with BN-Inception as the backbone by 0.8% and 1.1% on the UCF101 and HMDB51 datasets respectively, where both the methods uniformly sample 25 frames from each video for testing.…”

Section: On the Ucf101 And Hmdb51 Datasetsmentioning

confidence: 99%

Interaction-Aware Spatio-Temporal Pyramid Attention Networks for Action Classification

Liu

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

For CNN-based visual action recognition, the accuracy may be increased if local key action regions are focused on. The task of self-attention is to focus on key features and ignore irrelevant information. So, self-attention is useful for action recognition. However, the current self-attention methods usually ignore correlations among local feature vectors at spatial positions in feature maps in CNNs. In this paper, we propose an effective interaction-aware self-attention model which can extract information about the interactions between feature vectors to learn attention maps. Since the different layers in a network capture feature maps at different scales, we introduce a spatial pyramid with the feature maps at different layers to attention modeling. The multi-scale information is utilized to obtain more accurate attention scores. These attention scores are used to weight the local feature vectors of the feature maps and then calculate the attention feature maps. Since the number of feature maps input to the spatial pyramid attention layer is unrestricted, we easily extend this attention layer to a spatio-temporal version. Our model can be embedded into any general CNN to form a video-level end-to-end attention network for action recognition. Besides using the RGB stream alone, several methods are investigated to combine the RGB and flow streams for the final prediction of the classes of human actions. Experimental results show that our method achieves state-of-the-art results on the datasets UCF101, HMDB51, Kinetics-400, and untrimmed Charades.

show abstract

Section: On the Ucf101 And Hmdb51 Datasetsmentioning

confidence: 99%

Interaction-Aware Spatio-Temporal Pyramid Attention Networks for Action Classification

Liu

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Table 1 shows the results of all the 3D-CNN models when processing the new standard sequence images. All the models are from [38]. From the comparisons, we find that although the model parameters of ShuffNet are very limited, ResNet50 achieves the best AUC value in the test set.…”

Section: Comparison With the Classic 3d Cnn Networkmentioning

confidence: 99%

Semi-Automatic Classification of Prostate Cancer from Multi-Parametric MR Imaging Using Novel Image Sequences and 3D Convolutional Neural Networks

B¹,

Oka²,

Xuan³

et al. 2021

JCMI

View full text Add to dashboard Cite

“…In [17], the effect of dataset size on performance is investigated for several 3D-CNN architectures. Inflated versions of popular resourceefficient 2D-CNN architectures are analyzed for video classification tasks in [24]. In this work, we use the variants of 3D-CNNs for the AV-ASD task.…”

Section: Audio-visual Feature Extractionmentioning

confidence: 99%

“…Capturing motion patterns is crucial since movements of facial muscles and mouth are indicative of active speaking. We experimented with several high-performance and resource-efficient 3D-CNN architectures [24]. 3D-ResNeXt-101 performs best and becomes our final choice as video backbone.…”

Section: Audio-visual Encoder Architecturementioning

confidence: 99%

“…One can attribute the performance boost achieved by 3D-ResNet-18 backbone to its increased number of parameters and FLOPs. Therefore, we have used several resource efficient 3D CNNs [24] as video backbone and reported their performance at the bottom part of Table 3. It can be seen that all 3D CNN architectures achieve better performance than 2D-ResNet-18.…”

Section: Audio-visual Encoder Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild

Köpüklü¹,

Taseska²,

Rigoll³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Successful active speaker detection requires a threestage pipeline: (i) audio-visual encoding for all speakers in the clip, (ii) inter-speaker relation modeling between a reference speaker and the background speakers within each frame, and (iii) temporal modeling for the reference speaker. Each stage of this pipeline plays an important role for the final performance of the created architecture. Based on a series of controlled experiments, this work presents several practical guidelines for audio-visual active speaker detection. Correspondingly, we present a new architecture called ASDNet, which achieves a new state-of-the-art on the AVA-ActiveSpeaker dataset with a mAP of 93.5% outperforming the second best with a large margin of 4.7%. Our code and pretrained models are publicly available 1 .

show abstract

Resource Efficient 3D Convolutional Neural Networks

Cited by 164 publications

References 17 publications

Interaction-Aware Spatio-Temporal Pyramid Attention Networks for Action Classification

Interaction-Aware Spatio-Temporal Pyramid Attention Networks for Action Classification

Semi-Automatic Classification of Prostate Cancer from Multi-Parametric MR Imaging Using Novel Image Sequences and 3D Convolutional Neural Networks

How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild

Contact Info

Product

Resources

About