Spatio-Temporal AutoEncoder for Video Anomaly Detection

Zhao, Yiru; Deng, Bing; Shen, Chen; Liu, Yao; Lu, Hongtao; Hua, Xian-Sheng

doi:10.1145/3123266.3123451

Cited by 437 publications

(242 citation statements)

References 18 publications

Supporting

Mentioning

219

Contrasting

Unclassified

Order By: Relevance

“…Hasan et al [9] detect the anomalies according to the reconstruction error of a convolutional AE. Zhao et al [44] proposed to use 3D convolution based reconstruction and prediction. Luo et al [24] iteratively update the sparse coefficients via a stacked RNN to detect anomalies in videos.…”

Section: Related Workmentioning

confidence: 99%

Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection

Gong

Liu

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

1,126

686

View full text Add to dashboard Cite

Deep autoencoder has been extensively used for anomaly detection. Training on the normal data, the autoencoder is expected to produce higher reconstruction error for the abnormal inputs than the normal ones, which is adopted as a criterion for identifying anomalies. However, this assumption does not always hold in practice. It has been observed that sometimes the autoencoder "generalizes" so well that it can also reconstruct anomalies well, leading to the miss detection of anomalies. To mitigate this drawback for autoencoder based anomaly detector, we propose to augment the autoencoder with a memory module and develop an improved autoencoder called memory-augmented autoencoder, i.e. MemAE. Given an input, MemAE firstly obtains the encoding from the encoder and then uses it as a query to retrieve the most relevant memory items for reconstruction. At the training stage, the memory contents are updated and are encouraged to represent the prototypical elements of the normal data. At the test stage, the learned memory will be fixed, and the reconstruction is obtained from a few selected memory records of the normal data. The reconstruction will thus tend to be close to a normal sample. Thus the reconstructed errors on anomalies will be strengthened for anomaly detection. MemAE is free of assumptions on the data type and thus general to be applied to different tasks. Experiments on various datasets prove the excellent generalization and high effectiveness of the proposed MemAE.

show abstract

Section: Related Workmentioning

confidence: 99%

Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection

Gong

Liu

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

1,126

686

View full text Add to dashboard Cite

show abstract

“…Since the spatial appearances and temporal relations are both very important cues for video understanding, our work investigates the use of both spatial and temporal dimensions using 3D CNNs in videos. To our best knowledge, only few works (Zhao et al 2017;Vondrick, Pirsiavash, and Torralba 2016) exploit the 3D architectures in self-supervised feature learning. They use reconstruction/generation-based pretext tasks, and aim for a specific target task: anomaly detection and video generation respectively.…”

Section: Self-supervised Representation Learningmentioning

confidence: 99%

“…Since there are few prior works on self-supervised representation learning using 3D CNNs, we enumerate several alternative self-supervision tasks to provide our own reference levels and validate the effectiveness of our method. While we mainly focus on the context-based approaches, we also explore the reconstruction-based methods: spatio-temporal autoencoders (Zhao et al 2017) and 3D inpainting (Pathak et al 2016) as well. All the methods and experiments use the same 3D ResNet-18 as a backbone architecture, and use Kinetics dataset (without labels).…”

Section: Alternative Pretraining Strategiesmentioning

confidence: 99%

Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles

Kim

Cho

Kweon

2019

AAAI

339

238

View full text Add to dashboard Cite

Self-supervised tasks such as colorization, inpainting and zigsaw puzzle have been utilized for visual representation learning for still images, when the number of labeled images is limited or absent at all. Recently, this worthwhile stream of study extends to video domain where the cost of human labeling is even more expensive. However, the most of existing methods are still based on 2D CNN architectures that can not directly capture spatio-temporal information for video applications. In this paper, we introduce a new self-supervised task called as Space-Time Cubic Puzzles to train 3D CNNs using large scale video dataset. This task requires a network to arrange permuted 3D spatio-temporal crops. By completing Space-Time Cubic Puzzles, the network learns both spatial appearance and temporal relation of video frames, which is our final goal. In experiments, we demonstrate that our learned 3D representation is well transferred to action recognition tasks, and outperforms state-of-the-art 2D CNN-based competitors on UCF101 and HMDB51 datasets.

show abstract

“…Recent approaches rejuvenate the field by using convolutional neural networks (CNN) to extract high-level features from video frame intensity and achieve improved results. Some of these methods include Convolutional autoencoder [13], spatio-temporal autoencoder [5], 3D Convnet AE [27], and Temporally-coherent Sparse Coding Stacked-RNN [20]. Acknowledging the limitation of the intensity based features such as sensitivity to appearance noise, Liu et al [18] proposed to use the prediction of optical flow in their temporal coherent loss, effectively filtering out parts of the noise in pixel appearance.…”

Section: Video Anomaly Detectionmentioning

confidence: 99%

Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos

Morais

Tran

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

254

197

View full text Add to dashboard Cite

Appearance features have been widely used in video anomaly detection even though they contain complex entangled factors. We propose a new method to model the normal patterns of human movements in surveillance video for anomaly detection using dynamic skeleton features. We decompose the skeletal movements into two sub-components: global body movement and local body posture. We model the dynamics and interaction of the coupled features in our novel Message-Passing Encoder-Decoder Recurrent Network. We observed that the decoupled features collaboratively interact in our spatio-temporal model to accurately identify human-related irregular events from surveillance video sequences. Compared to traditional appearancebased models, our method achieves superior outlier detection performance. Our model also offers "open-box" examination and decision explanation made possible by the semantically understandable features and a network architecture supporting interpretability.

show abstract

Spatio-Temporal AutoEncoder for Video Anomaly Detection

Cited by 437 publications

References 18 publications

Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection

Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection

Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles

Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos

Contact Info

Product

Resources

About