Spatio-Temporal Unity Networking for Video Anomaly Detection

Abstract

Anomaly detection in video surveillance is challenging due to the variety of anomaly types and definitions, which limit the use of supervised techniques. As such, auto-encoder structures, a type of classical unsupervised method, have recently been utilized in this field. These structures consist of an encoder followed by a decoder and are typically adopted to restructure a current input frame or predict a future frame. However, regardless of whether a 2D or 3D autoencoder structure is adopted, only single-scale information from the previous layer is typically used in the decoding process. This can result in a loss of detail that could potentially be used to predict or reconstruct video frames. As such, this study proposes a novel spatio-temporal U-Net for frame prediction using normal events and abnormality detection using prediction error. This framework combines the benefits of U-Nets in representing spatial information with the capabilities of ConvLSTM for modeling temporal motion data. In addition, we propose a new regular score function, consisting of a prediction error for not only the current frame but also future frames, to further improve the accuracy of anomaly detection. Extensive experiments on common anomaly datasets, including UCSD (98 video clips in total) and CUHK Avenue (30 video clips in total), validated the performance of the proposed technique and we achieved 96.5% AUC for the Ped2 dataset, which is much better than existing autoencoder-based and U-Net-based methods.

Spatio-temporal prediction and reconstruction network for video anomaly detection

“…The input pattern of our model is different from the current methods that stack T sequential frames together into the model. Among these methods, the T frames are linked to each corresponding channel in the first output feature data, resulting in the collapse of temporal information [ 22 ]. Thus, we feed T frames into the encoder orderly to generate corresponding feature maps.…”

Integrated Multiscale Appearance Features and Motion Information Prediction Network for Anomaly Detection

“…The input mode of our network is different from existing methods that conventionally stack T consecutive frames together into a network. In these methods, all the T frames are connected to each channel in the first output feature map, which results in the collapse of temporal information [ 29 ]; thus, we input T frames into the encoder network one by one to generate corresponding feature maps. As shown in Figure 4 , the DB-ConvLSTM structure includes a shallow forward layer and a deeper backward layer.…”

Adaptive Adversarial Latent Space for Novelty Detection

“…During testing, two networks take each frame as input and the outputs jointly determine whether it is novel or not. Li et al [29] propose a new anomaly score function and a spatio-temporal framework combined by U-net and adversarial learning. Similarly, Dong et al [30] propose a new approach with a dual discriminator-based generative adversarial network and U-net structure.…”