Mining Persistent Activity in Continually Evolving Networks

Abstract

Frequent pattern mining is a key area of study that gives insights into the structure and dynamics of evolving networks, such as social or road networks. However, not only does a network evolve, but often the way that it evolves, itself evolves. Thus, knowing, in addition to patterns' frequencies, for how long and how regularly they have occurred-i.e., their persistence-can add to our understanding of evolving networks. In this work, we propose the problem of mining activity that persists through time in continually evolving networks-i.e., activity that repeatedly and consistently occurs. We extend the notion of temporal motifs to capture activity among specific nodes, in what we call activity snippets, which are small sequences of edge-updates that reoccur. We propose axioms and properties that a measure of persistence should satisfy, and develop such a persistence measure. We also propose PENminer, an efficient framework for mining activity snippets' Persistence in Evolving Networks, and design both offline and streaming algorithms. We apply PENminer to numerous real, large-scale evolving networks and edge streams, and find activity that is surprisingly regular over a long period of time, but too infrequent to be discovered by aggregate count alone, and bursts of activity exposed by their lack of persistence. Our findings with PENminer include neighborhoods in NYC where taxi traffic persisted through Hurricane Sandy, the opening of new bike-stations, characteristics of social network users, and more. Moreover, we use PENminer towards identifying anomalies in multiple networks, outperforming baselines at identifying subtle anomalies by 9.8-48% in AUC.

Sketch-Based Anomaly Detection in Streaming Graphs

“…SEDANSPOT [11] identifies edge anomalies based on edge occurrence, preferential attachment, and mutual neighbors. PENminer [12] explores the persistence of activity snippets, i.e., the length and regularity of edge-update sequences' reoccurrences. F-FADE [13] aims to detect anomalous interaction patterns by factorizing their frequency.…”

“…We use open-source implementations of DENSESTREAM [1] (Java), SEDANSPOT [11] (C++), MIDAS-R [3] (C++), PENminer [12] (Python), F-FADE [13] (Python), DENSEALERT [1] (Java), and ANOMRANK [14] (C++) provided by the authors, following parameter settings as suggested in the original paper. For SPOTLIGHT [2], we used open-sourced implementations of Random Cut Forest [55] and Carter Wegman hashing [56].…”

“…However, these approaches focus on static graphs, whereas many real-world graphs are time-evolving in nature. In streaming or online graph scenarios, some methods can detect the presence of anomalous edges, [3,[11][12][13], while others can detect anomalous subgraphs [1,2,14]. However, all existing methods are limited to either anomalous edge or graph detection but not able to detect both kinds of anomalies, as summarized in Table 1.…”

Isconna: Streaming Anomaly Detection with Frequency and Patterns

“…PENminer [8] explores the persistence of activity snippets, i.e., the length and regularity of edge-update sequences' reoccurrences. F-FADE [15] aims to detect anomalous interaction patterns by factorizing the frequency of those patterns.…”

“…On the other hand, there is another category of algorithms that aims to detect anomalies using patterns or motifs [8,15,43,38,30,33,55,3,23,29,41,40,35,31,7,20,11]. However, many of these methods require active exploration of patterns or snippets, increasing memory and time requirements.…”