Fast anomaly detection despite the duplicates

Lee, Jay Yoon; Kang, U; Koutra, Danai; Faloutsos, Christos

doi:10.1145/2487788.2487886

Cited by 11 publications

(7 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given a cloud of multi-dimensional points, GFADD [15] detects outliers in a scalable way by taking care of the major problem of duplicate points. Fast Anomaly Detection given Duplicates (FADD) solves duplicate problems by treating them as one super node rather than considering them separately.…”

Section: Grid-based Fast Anomaly Detection Given Duplicates (Gfadd)mentioning

confidence: 99%

Detecting insider threats in a real corporate database of computer usage activity

Senator

Goldberg

Memory

et al. 2013

Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Self Cite

View full text Add to dashboard Cite

This paper reports on methods and results of an applied research project by a team consisting of SAIC and four universities to develop, integrate, and evaluate new approaches to detect the weak signals characteristic of insider threats on organizations' information systems. Our system combines structural and semantic information from a real corporate database of monitored activity on their users' computers to detect independently developed red team inserts of malicious insider activities. We have developed and applied multiple algorithms for anomaly detection based on suspected scenarios of malicious insider behavior, indicators of unusual activities, high-dimensional statistical patterns, temporal sequences, and normal graph evolution. Algorithms and representations for dynamic graph processing provide the ability to scale as needed for enterpriselevel deployments on real-time data streams. We have also developed a visual language for specifying combinations of features, baselines, peer groups, time periods, and algorithms to detect anomalies suggestive of instances of insider threat behavior. We defined over 100 data features in seven categories based on approximately 5.5 million actions per day from approximately 5,500 users. We have achieved area under the ROC curve values of up to 0.979 and lift values of 65 on the top 50 user-days identified on two months of real data.

show abstract

Section: Grid-based Fast Anomaly Detection Given Duplicates (Gfadd)mentioning

confidence: 99%

Detecting insider threats in a real corporate database of computer usage activity

Senator

Goldberg

Memory

et al. 2013

Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Self Cite

View full text Add to dashboard Cite

show abstract

“…Anomaly detection in static graphs has been studied using various data mining and statistical techniques [11,19,22,34]. Two surveys [35,36] describe various outlier detection methods for static and time-evolving graphs.…”

Section: Anomaly Detectionmentioning

confidence: 99%

“…Local outlier factor (LOF) [37] is a popular density-based anomaly detection method that compares the local density of each data point to the density of its neighbors by employing the k-nearest neighbor (kNN) technique to measure how isolated an object is with respect to its neighborhood. Grid-based FADD (G-FADD) [38] improves on the quadratic complexity of LOF by cleverly handling duplicate points and applying a grid on a multi-dimensional space so that only cells that satisfy density-specific criteria need investigation for outliers. AutoPart [39] is a graph clustering algorithm that detects anomalous edges based on the Minimum Description Language (MDL) encoding principle.…”

Section: Anomaly Detectionmentioning

confidence: 99%

“…For this purpose, the system leverages algorithms such as G-FADD (Grid Fast Anomaly Detection algorithm given Duplicates [11]), a fast, density-based anomaly detection method to find local and global outliers in the distribution plots, as well as ABOD (Angle Based Outlier Detection [12]), an angle-based outlier detection method that is particularly suitable for analyzing highly dimensional data.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

PERSEUS-HUB: Interactive and Collective Exploration of Large-Scale Graphs

et al. 2017

Self Cite

View full text Add to dashboard Cite

Abstract:Graphs emerge naturally in many domains, such as social science, neuroscience, transportation engineering, and more. In many cases, such graphs have millions or billions of nodes and edges, and their sizes increase daily at a fast pace. How can researchers from various domains explore large graphs interactively and efficiently to find out what is 'important'? How can multiple researchers explore a new graph dataset collectively and "help" each other with their findings? In this article, we present PERSEUS-HUB, a large-scale graph mining tool that computes a set of graph properties in a distributed manner, performs ensemble, multi-view anomaly detection to highlight regions that are worth investigating, and provides users with uncluttered visualization and easy interaction with complex graph statistics. PERSEUS-HUB uses a Spark cluster to calculate various statistics of large-scale graphs efficiently, and aggregates the results in a summary on the master node to support interactive user exploration. In PERSEUS-HUB, the visualized distributions of graph statistics provide preliminary analysis to understand a graph. To perform a deeper analysis, users with little prior knowledge can leverage patterns (e.g., spikes in the power-law degree distribution) marked by other users or experts. Moreover, PERSEUS-HUB guides users to regions of interest by highlighting anomalous nodes and helps users establish a more comprehensive understanding about the graph at hand. We demonstrate our system through the case study on real, large-scale networks.

show abstract

“…Traditional techniques of outlier or anomaly detection are based on local density estimaties such as knearest neighbors. Examples include Local Outlier Factor (LOF) [3] and its numerous variants [21,18,19,11].…”

Section: Outlier Detection Outliers As Per Barnett Andmentioning

confidence: 99%

On Skewed Multi-dimensional Distributions: the FusionRP Model, Algorithms, and Discoveries

Pillutla

Fang

Devineni

et al. 2016

Proceedings of the 2016 SIAM International Conference on Data Mining

Self Cite

View full text Add to dashboard Cite

How do we model and find outliers in Twitter data? Given the number of retweets of each person on a social network, what is their expected number of comments? Real-life data are often very skewed, exhibiting power-law-like behavior. For such skewed multidimensional discrete data, the existing models are not general enough to capture various realistic scenarios, and need to be discretized as they often model continuous quantities. We propose FusionRP, short for Fusion Restaurant Process, a simple and intuitive model for skewed multi-dimensional discrete distributions, such as number of retweets vs. comments in Twitter-like data. Our model is discrete by design, has provably asymptotic log-logistic sum of marginals , is general enough to capture varied relationships, and most importantly, fits real data very well. We give an effective and scalable maximum-likelihood based fitting approach that is linear in the number of unique observed values and the input dimension. We test FusionRP on a twitter-like social network with 2.2M users, a phone call network with 1.9M call records, game data with 45M users and Facebook data with 2.5M posts. Our results show that FusionRP significantly outperforms several alternative methods and can detect outliers, such as bot-like behaviors in the Facebook data.

show abstract

Fast anomaly detection despite the duplicates

Cited by 11 publications

References 6 publications

Detecting insider threats in a real corporate database of computer usage activity

Detecting insider threats in a real corporate database of computer usage activity

PERSEUS-HUB: Interactive and Collective Exploration of Large-Scale Graphs

On Skewed Multi-dimensional Distributions: the FusionRP Model, Algorithms, and Discoveries

Contact Info

Product

Resources

About