Composable, scalable, and accurate weight summarization of unaggregated data sets

Abstract

Many data sets occur as unaggregated data sets, where multiple data points are associated with each key. In the aggregate view of the data, the weight of a key is the sum of the weights of data points associated with the key. Examples are measurements of IP packet header streams, distributed data streams produced by events registered by sensor networks, and Web page or multimedia requests to context distribution servers. We aim to combine sampling and aggregation to provide accurate and efficient summaries of the aggregate view. However, data points are scattered in time or across multiple servers and hence aggregation is subject to resource constraints on the size of summaries that can be stored or transmitted.We develop a summarization framework for unaggregated data where summarization is a scalable and composable operator, and as such, can be tailored to meet resource constraints. Our summaries support unbiased estimates of the weight of subpopulations of keys specified using arbitrary selection predicates. While we prove that under such scenarios there is no variance optimal scheme, our estimators have the desirable properties that the variance is progressively closer to the minimum possible when applied to a "more" aggregated data set. An extensive evaluation using synthetic and real data sets shows that our summarization framework outperforms all existing schemes for this fundamental problem, even for the special and well-studied case of data streams.

Integration Of Big Data And Cloud Computing To Detect Black Money Check Rotation With Range Aggregate Queries

“…Haas and K€onig€ [23] proposed a new sampling scheme, which combines the rowlevel and page-level samplings in the field of relational DBMS. Data sampling is also well used in the field of distributed and streaming environments [24], [25]. Histogram is another important technique for selectivity estimation.…”

Stream Sampling for Frequency Cap Statistics

“…There is a large body of work on computing statistics over unaggregated data which we can not hope to cover here. The toolbox includes deterministic algorithms [29], other sampling algorithms [8], and Linear sketches (random linear projections) [26], [1], [25], [13] Deterministic algorithms work well for approximate heavy hitters and quantiles. Linear sketches project the key-weight vectors to a lower dimensional vector.…”

Count Me In: Viable Distributed Summary Statistics for Securing High-Speed Networks

“…In this example, val\]func simply returns the current number of connection attempts for a host. 4 However, the function could be more complex than that. In our application, one could for example instead implement a threshold relative to the number of successful connections.…”

“…Cohen et al [4] present an abstract framework for weighted sampling in distributed settings. It is similar in intent to our work, however, it only considers the case of sampling, and evaluates optimal algorithms for this setting.…”