Weakly-Supervised Spatio-Temporal Anomaly Detection in Surveillance Video

Abstract

In this paper, we introduce a novel task, referred to as Weakly-Supervised Spatio-Temporal Anomaly Detection (WSSTAD) in surveillance video. Specifically, given an untrimmed video, WSSTAD aims to localize a spatio-temporal tube (i.e., a sequence of bounding boxes at consecutive times) that encloses the abnormal event, with only coarse video-level annotations as supervision during training. To address this challenging task, we propose a dual-branch network which takes as input the proposals with multi-granularities in both spatial-temporal domains. Each branch employs a relationship reasoning module to capture the correlation between tubes/videolets, which can provide rich contextual information and complex entity relationships for the concept learning of abnormal behaviors. Mutually-guided Progressive Refinement framework is set up to employ dual-path mutual guidance in a recurrent manner, iteratively sharing auxiliary supervision information across branches. It impels the learned concepts of each branch to serve as a guide for its counterpart, which progressively refines the corresponding branch and the whole framework. Furthermore, we contribute two datasets, i.e., ST-UCF-Crime and STRA, consisting of videos containing spatio-temporal abnormal annotations to serve as the benchmarks for WSSTAD. We conduct extensive qualitative and quantitative evaluations to demonstrate the effectiveness of the proposed approach and analyze the key factors that contribute more to handle this task.

Detection of Fights in Videos: A Comparison Study of Anomaly Detection and Action Recognition

“…Sultani et al [22] propose the MIL framework using only video-level labels and introduce the large-scale anomaly detection dataset, UCF-Crime. This work inspires quite a few follow-up studies [28], [17], [4], [14], [26], [25], [6], [23], [26]. .…”

A Three-Stage Anomaly Detection Framework for Traffic Videos

“…Waqas et al [41] applied pretrained 3D networks to extract spatiotemporal features and trained the classifier with multi-instance learning techniques. Following this work, Zhu and Newsam [45] introduced optical flow; Lin et al [53] proposed a dual-branch network; Lv et al [54] replaced the feature extractor with a TSN and proposed an HCE module to capture dynamic changes; Feng et al [55] applied pseudolabel and self-attentive feature encoders for training; Wu et al [56] also proposed a dual-branch network but with tubular and temporal branches and so on.…”

An adaptive algorithm based on principal component analysis-deep learning for anomalous events detection

“…Another method for handling anomaly detection and classification using a weakly supervised learning model was provided by Majhi et al [16]. Employing multi-detail ideas in both the temporal and spatial dimensions as input, a dual branch network has been developed by Wu et al [17]. To detect video anomalies, Cao et al [18] suggested taking into consideration the spatial-temporal relationships between video parts.…”