Not only Look, But Also Listen: Learning Multimodal Violence Detection Under Weak Supervision

Wu, Peng; Liu, Jing; Shi, Yujia; Sun, Yujia; Shao, Fangtao; Wu, Zhaoyang; Yang, Zhiwei

doi:10.1007/978-3-030-58577-8_20

Cited by 186 publications

(160 citation statements)

References 45 publications

Supporting

Mentioning

158

Contrasting

Unclassified

Order By: Relevance

“…Modifying the proposed neural network to enhance its real-time characteristic is a direction for future work. [32] 30.77 Sultani et al [12] 73.20 Wu et al [15] 78.64 Ours (master branch) 81.28 Ours (combination) 81.69…”

Section: Discussionmentioning

confidence: 99%

“…The overall architecture of our proposed neural network is shown in Figure 1. Following [15], visual features and audio features are first extracted from the pretrained models I3D [18] and VGGish [19], respectively. After that, the two types of features are fused by three different modules shown in Figure 1.…”

Section: Methodsmentioning

confidence: 99%

“…In this study, we treat the violence detection task as a weakly supervised detection problem where only video-level labels are offered in the training set, and frame-level labels prediction is required by the task, and MIL is applied in the training phase. Following [12,15], we take each video as a bag and each video segment as an instance; a video including any violence is a positive bag (labeled as 1), and a video excluding violence is a negative bag (labeled as 0). In this study, K instances with the highest scores are employed to calculate the training loss with the MSE function:…”

Section: Training and Determination Methodsmentioning

confidence: 99%

“…Parameters  v and  a in the co-attention module are set as 1. Notably, for fair evaluation, visual and audio features utilized are the same as [15], which are extracted from a pretrained I3D model and VGGish model. During the testing phase, frame-level average precision (AP) is utilized as the evaluation metric.…”

Section: Implementation Detailsmentioning

confidence: 99%

“…Sultani et al [12] applied visual features extracted from a 3D-convolutional network (C3D) [13] and multiple instance learning (MIL) [14] in their proposed method. Recently, [15] built a large-scale dataset XD-Violence for multimodal learning in the violence detection task. They also proposed a HL-Net leveraging self-attention [16] and GCN [17].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Violence Detection in Videos Based on Fusing Visual and Audio Information

Pang

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Determining whether given video frames contain violent content is a basic problem in violence detection. Visual and audio information are useful for detecting violence included in a video, and are usually complementary; however, violence detection studies focusing on fusing visual and audio information are relatively rare. Therefore, we explored methods for fusing visual and audio information. We proposed a neural network containing three modules for fusing multimodal information: 1) attention module for utilizing weighted features to generate effective features based on the mutual guidance between visual and audio information; 2) fusion module for integrating features by fusing visual and audio information based on the bilinear pooling mechanism; and 3) mutual Learning module for enabling the model to learn visual information from another neural network with a different architecture. Experimental results indicated that the proposed neural network outperforms existing state-of-the-art methods on the XD-Violence dataset.

show abstract

Section: Discussionmentioning

confidence: 99%