Sketch ⋆-Metric: Comparing Data Streams via Sketching

Abstract

In this paper, we consider the problem of estimating the distance between any two large data streams in smallspace constraint. This problem is of utmost importance in data intensive monitoring applications where input streams are generated rapidly. These streams need to be processed on the fly and accurately to quickly determine any deviance from nominal behavior. We present a new metric, the Sketch ⋆-metric, which allows to define a distance between updatable summaries (or sketches) of large data streams. An important feature of the Sketch ⋆-metric is that, given a measure on the entire initial data streams, the Sketch ⋆-metric preserves the axioms of the latter measure on the sketch (such as the non-negativity, the identity, the symmetry, the triangle inequality but also specific properties of the f -divergence). Extensive experiments conducted on both synthetic traces and real data allow us to validate the robustness and accuracy of the Sketch ⋆-metric.

A Sketch-based Clustering Algorithm for Uncertain Data Streams

“…Papapetrou et al present a novel sketching technique ECM-sketch that allows effective summarization of distributed data streams over sliding windows with probabilistic accuracy guarantees [16]. For measuring the distance between updatable sketches, Anceaume et al present a novel metric Sketch *-metric that reflects the relationships between any two discrete probability distributions in the massive data streams [3].…”

“…Since the data of objects are stored in the EU-sketches, the computation of similarity should be designed based on the EU-sketch. We combine the Sketch *-metric [3] and the KL divergence [2] to quantify the similarity between two uncertain objects.…”

“…To compute the Sketch-metric of two objects p and q, two EU-sketches UES p and UES q are constructed. As the sketch *-metric algorithm defined in [3], each line of the EU-sketch corresponds to a ρ i partition, 1≤i≤c. Thus the similarity between two objects is the maximal value over all the c partitions ρ i of the distance metric ϕ.…”

“…We use hash technique to compress data of objects to extended uncertain sketches, and update the data in sketches. Then, based on the KL divergence [2] and sketch *-metric [3], we present our sketch based similarity for objects distances measure. And then according the average probabilities value in sketches, we map objects to different core-sets.…”