Dynamic interaction graphs with probabilistic edge decay

Abstract

Abstract-A large scale network of social interactions, such as mentions in Twitter, can often be modeled as a "dynamic interaction graph" in which new interactions (edges) are continually added over time. Existing systems for extracting timely insights from such graphs are based on either a cumulative "snapshot" model or a "sliding window" model. The former model does not sufficiently emphasize recent interactions. The latter model abruptly forgets past interactions, leading to discontinuities in which, e.g., the graph analysis completely ignores historically important influencers who have temporarily gone dormant. We introduce TIDE, a distributed system for analyzing dynamic graphs that employs a new "probabilistic edge decay" (PED) model. In this model, the graph analysis algorithm of interest is applied at each time step to one or more graphs obtained as samples from the current "snapshot" graph that comprises all interactions that have occurred so far. The probability that a given edge of the snapshot graph is included in a sample decays over time according to a user specified decay function. The PED model allows controlled trade-offs between recency and continuity, and allows existing analysis algorithms for static graphs to be applied to dynamic graphs essentially without change. For the important class of exponential decay functions, we provide efficient methods that leverage past samples to incrementally generate new samples as time advances. We also exploit the large degree of overlap between samples to reduce memory consumption from O(N ) to O(log N ) when maintaining N sample graphs. Finally, we provide bulk-execution methods for applying graph algorithms to multiple sample graphs simultaneously without requiring any changes to existing graph-processing APIs. Experiments on a real Twitter dataset demonstrate the effectiveness and efficiency of our TIDE prototype, which is built on top of the Spark distributed computing framework.

General Temporally Biased Sampling Schemes for Online Model Management

“…Periodic data patterns can lead to the same phenomenon. Another example, from [35], concerns in uencers on Twitter: a proli c tweeter might temporarily stop tweeting due to travel, illness, or some other reason, and hence be completely forgotten in a sliding-window approach. Indeed, in real-world Twitter data, almost a quarter of top in uencers were of this type, and were missed by a sliding window approach.…”

“…By using a time-biased sample, the retraining costs can be held to an acceptable level while not sacri cing robustness in the presence of recurrent patterns. This approach was proposed in [35] in the setting of graph analysis algorithms, and has recently been adopted in the MacroBase system [3]. The orthogonal problem of choosing when to retrain a model is also an important question, and is related to, e.g., the literature on "concept drift" [15]; in this paper we focus on the problem of how to e ciently maintain a time-biased sample.…”

“…The authors therefore argue for supporting a rich class of decay functions. Similarly, Xie et al [35] show how a decay function can be chosen to meet application-speci c criteria. For example, by using an exponential decay function f (α) = exp(−λα) with λ = 0.058, a data item from 40 batches ago is 1/10 as likely to appear in the current analysis as a newly arrived item.…”

“…T-TBS: We rst provide and analyze Targeted-Size Time-Biased Sampling (T-TBS), a relatively simple algorithm that generalizes the Bernoulli sampling scheme in [35]. T-TBS allows complete control over the decay rate (expressed in wall-clock time) and probabilistically maintains a target sample size.…”

Big Graph Analytics Platforms

“…However, real-world graphs often evolve over time, with vertices and edges continually being added or deleted, and their attributes being frequently updated. A new class of big graph systems, such as KineoGraph [Cheng et al, 2012], TIDE [Xie et al, 2015b], DeltaGraph [Khurana and Deshpande, 2013], and Chronos [Han et al, 2014b], have emerged to process and analyze temporal and streaming graph data. This area is however still in its infancy and there are many open problems that need to be addressed to effectively handle continuous and/or temporal analytics on big graphs.…”