Real-World Anomaly Detection in Surveillance Videos

Abstract

Surveillance videos are able to capture a variety of realistic anomalies. In this paper, we propose to learn anomalies by exploiting both normal and anomalous videos. To avoid annotating the anomalous segments or clips in training videos, which is very time consuming, we propose to learn anomaly through the deep multiple instance ranking framework by leveraging weakly labeled training videos, i.e. the training labels (anomalous or normal) are at videolevel instead of clip-level. In our approach, we consider normal and anomalous videos as bags and video segments as instances in multiple instance learning (MIL), and automatically learn a deep anomaly ranking model that predicts high anomaly scores for anomalous video segments. Furthermore, we introduce sparsity and temporal smoothness constraints in the ranking loss function to better localize anomaly during training.We also introduce a new large-scale first of its kind dataset of 128 hours of videos. It consists of 1900 long and untrimmed real-world surveillance videos, with 13 realistic anomalies such as fighting, road accident, burglary, robbery, etc. as well as normal activities. This dataset can be used for two tasks. First, general anomaly detection considering all anomalies in one group and all normal activities in another group. Second, for recognizing each of 13 anomalous activities. Our experimental results show that our MIL method for anomaly detection achieves significant improvement on anomaly detection performance as compared to the state-of-the-art approaches. We provide the results of several recent deep learning baselines on anomalous activity recognition. The low recognition performance of these baselines reveals that our dataset is very challenging and opens more opportunities for future work. The dataset is available at: https

3D ResNet with Ranking Loss Function for Abnormal Activity Detection in Videos

“…The segments from abnormal videos have been grouped into positive instances, and the segments of the normal videos have been grouped into the negative instances. Where, C a and C n are the representation for positive group and negative group respectively [3].…”

“…Equations 3 and 4 are to avoid case 1 of the false alarm, when the model predicts a normal instance as an abnormal instance. Equation 3 compares the maximum ranked instances from each group [3], where the maximum ranked instance from the positive group is most likely to be the true positive and the maximum ranked instance from the negative group can be the case of false positive. Equation 4 compares the maximum ranked instance and minimum ranked instance from the positive group, where the maximum ranked instance from the positive group is most likely to be the true positive and the minimum ranked instance from the positive group can be the case of false positive.…”

“…where, l 1 (C a , C n ) , l 2 (C a , C a ) , l 3 (C a , C n ) and l 4 (C a , C n ) are defined as below: l 4 (C a , C n ) = max 0, 1 − M 3 (S (I a )) + max i∈Cn S I i n (12) As in the study of W. Sultani et al [3], to maintain the temporal smoothness of the abnormality score and to maintain the sparsity of the abnormal scores, two constraints have been proposed as below:…”

“…have been proposed to detect abnormal activities in videos. Recently, W. Sultani et al [3], has proposed a new approach to solve all the shortcomings of the traditional approaches. In their method, to detect the abnormal activity, they have trained their model with the help of multiple instance learning (MIL) [13], [14] by utilizing the weakly labeled training videos.…”

“…Where recent advancement of 3D deep neural network in the video action recognition task motivates us to use 3D ResNet [15], [16] in the abnormal activity detection task. Extracted features from 3D ResNet and video-level labeling are used to train the model in a weakly-supervised manner with the help of deep MIL [3], [13] along with the new proposed ranking loss.…”