Nowadays, the analysis of abnormal events becomes more and more exhausting due to the divine use of surveillance cameras. Reliability of normal and abnormal samples is generally different in practice. In the existing work, developed machine learning and swarm intelligence based approaches for anomaly detection by extracting spatiotemporal features from video sequences. This proposed abnormal event detection mechanism is obvious, but the tracking result is less precise. Some of the reasons are low quality video, system noise, small object, and other factors. In order to improve the precisions of the tracked object, this research work proposed a new hybrid deep learning and robust segmentation method for better and faster tracking result. In the proposed approach, a video frame is subjected to a series of operations to extract most salient information from it. Initially, a 2D variance plane is constructed to encode local spatio-temporal variations around each pixel in a video frame. The Improved Particle Swarm Optimization algorithm is then applied to isolate the most salient regions based on motion information in the 2D variance plane. Grey Level Co-occurrence Matrix is applied to the extracted salient pixels in the video. And then the segmentation process is done with the help of fuzzy c-means clustering of successive frames are exploited for pattern matching in a simple feature space. Finally developed a hybrid deep learning based on a pre-trained Convolution Neural Network and One-class SVM is trained with spatial features for robust classification of abnormal shapes. Thus the experimental result suggests that the proposed anomaly detection techniques outperform the existing techniques in context of accuracy and time complexity.