Distributed Outlier Detection using Compressive Sensing

Yan, Ying; Zhang, Jiaxing; Huang, Bojun; Sun, Xuzhan; Mu, Jiaqi; Zhang, Zheng; Moscibroda, Thomas

doi:10.1145/2723372.2747641

Cited by 10 publications

(4 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Liu et al [18] proposed an Entropy-based Fast Detection algorithm for outlier detection in large datasets. Yan et al [26] proposed a distributed outlier detection algorithm which employs compressive sensing for sampling high-dimensional data. Dimensionality Reduction for OUTlier detection (DROUT) can be used for anomaly detection in "very wide" datasets i.e., datasets with large number of features.…”

Section: Anomaly Detection Algorithmsmentioning

confidence: 99%

FAAD: A Self-Optimizing Algorithm for Anomaly Detection

Hashmi

Ahmad

2019

IAJIT

View full text Add to dashboard Cite

Anomaly/Outlier detection is the process of finding abnormal data points in a dataset or data stream. Most of the anomaly detection algorithms require setting of some parameters which significantly affect the performance of the algorithm. These parameters are generally set by hit-and-trial; hence performance is compromised with default or random values. In this paper, the authors propose a self-optimizing algorithm for anomaly detection based on firefly meta-heuristic, and named as Firefly Algorithm for Anomaly Detection (FAAD). The proposed solution is a non-clustering unsupervised learning approach for anomaly detection. The algorithm is implemented on Apache Spark for scalability and hence the solution can handle big data as well. Experiments were conducted on various datasets, and the results show that the proposed solution is much accurate than the standard algorithms of anomaly detection.

show abstract

Section: Anomaly Detection Algorithmsmentioning

confidence: 99%

FAAD: A Self-Optimizing Algorithm for Anomaly Detection

Hashmi

Ahmad

2019

IAJIT

View full text Add to dashboard Cite

show abstract

“…A related yet different problem is examined in [24]. In a production distributed environment, a stream of data points may be split across multiple nodes, each holding part of the values of a data point.…”

Section: Related Workmentioning

confidence: 99%

Parallel Continuous Outlier Mining in Streaming Data

Toliopoulos

Gounaris

Tsichlas

et al. 2018

2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA)

View full text Add to dashboard Cite

In this work, we focus on distance-based outliers in a metric space, where the status of an entity as to whether it is an outlier is based on the number of other entities in its neighborhood. In recent years, several solutions have tackled the problem of distance-based outliers in data streams, where outliers must be mined continuously as new elements become available. An interesting research problem is to combine the streaming environment with massively parallel systems to provide scalable streambased algorithms. However, none of the previously proposed techniques refer to a massively parallel setting. Our proposal fills this gap and investigates the challenges in transferring state-of-the-art techniques to Apache Flink, a modern platform for intensive streaming analytics. We thoroughly present the technical challenges encountered and the alternatives that may be applied. We show speed-ups of up to 117 (resp. 2076) times over a naive parallel (resp. non-parallel) solution in Flink, by using just an ordinary four-core machine and a real-world dataset. When moving to a three-machine cluster, due to less contention, we manage to achieve both better scalability in terms of the window slide size and the data dimensionality, and even higher speed-ups, e.g., by a factor of 510. Overall, our results demonstrate that oulier mining can be achieved in an efficient and scalable manner. The resulting techniques have been made publicly available as open-source software.

show abstract

“…Yan et al [31] formulated the bias recovery problem in the context of distributed outlier detection. We briefly describe how BOMP works.…”

Section: Related Workmentioning

confidence: 99%

Bias-aware sketches

Chen

Zhang

2017

Proc. VLDB Endow.

View full text Add to dashboard Cite

Linear sketching algorithms have been widely used for processing large-scale distributed and streaming datasets. Their popularity is largely due to the fact that linear sketches can be naturally composed in the distributed model and be efficiently updated in the streaming model. The errors of linear sketches are typically expressed in terms of the sum of coordinates of the input vector excluding those largest ones, or, the mass on the tail of the vector. Thus, the precondition for these algorithms to perform well is that the mass on the tail is small, which is, however, not always the case -in many real-world datasets the coordinates of the input vector have a bias, which will generate a large mass on the tail.In this paper we propose linear sketches that are biasaware. We rigorously prove that they achieve strictly better error guarantees than the corresponding existing sketches, and demonstrate their practicality and superiority via an extensive experimental evaluation on both real and synthetic datasets.

show abstract

Distributed Outlier Detection using Compressive Sensing

Cited by 10 publications

References 45 publications

FAAD: A Self-Optimizing Algorithm for Anomaly Detection

FAAD: A Self-Optimizing Algorithm for Anomaly Detection

Parallel Continuous Outlier Mining in Streaming Data

Bias-aware sketches

Contact Info

Product

Resources

About