Violence recognition on videos using two-stream 3D CNN with custom spatiotemporal crop

Abstract

Violence may happen anywhere. One of the ways to know and oversee the violence in some places is by installing Closed-circuit Television (CCTV). The recorded video captured by CCTV can be used as proof in a law court. Violence video classification is also one of the topics being discussed in deep learning. The latest violence video dataset is RWF-2000. That dataset contains violent and non-violent videos, 5 seconds duration, 30 frames per second, with the amount of 2000 videos. That publication also has the best accuracy of 87.25% by their proposed method. In this study, we will use a Residual Network known to have the advantage of solving the vanishing gradient problem. Beside that, we also implement transfer learning from Kinetics and Kinetics + Moments in Time pre-trained data. We also test the number of frames and the location of the sampling frame range. RGB and optical flow inputs are separately trained with different configurations. The RGB input best accuracy is 89.25% with pre-trained Kinetics + Moments in Time, using frame location 49-149. The optical flow input best accuracy is 88.5% with pre-trained Kinetics, using 74 frames. We also try to sum the output of both inputs making accuracy of 90.5%.

KianNet: A Violence Detection Model Using an Attention-Based CNN-LSTM Structure

“…Finally, the output features from both streams are merged to make a final prediction. In a recent study, Pratama et al [27] proposed a two-stream 3D CNN model that uses RGB and optical flow images for VD.…”

“…Model Accuracy (%) Sudhakaran et al [40] Convolutional LSTM 77 Tran et al [22] C3D 82.75 Cheng et al [16] Flow Gated Net 87.25 Su et al [50] SPIL Convolution 89.3 Islam et al [51] SepConvLSTM-M 89.75 Pratama et al [27] Two-stream 3D CNN 90.50 Kang et al [52] 2D CNNs + LSTM 92 Chelali et al [53] 2D Spatio-Temporal 93.80 Magdy et al [23] Violence 4D 94.67 Proposed method KianNet 96.21…”