Space-Efficient Estimation of Statistics Over Sub-Sampled Streams

McGregor, Andrew; Pavan, A.; Tirthapura, Srikanta; Woodruff, David P.

doi:10.1007/s00453-015-9974-0

Cited by 10 publications

(5 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Continuous random sampling from the set of distinct elements in a stream has been considered in [26]. The question of how to process a "sampled stream", i.e., once a stream has been sampled, is considered in [27]. A model of distributed streams related to ours was considered in [28], [29].…”

Section: Related Workmentioning

confidence: 99%

A Simple Message-Optimal Algorithm for Random Sampling from a Distributed Stream

Chung

Tirthapura

Woodruff

2016

IEEE Trans. Knowl. Data Eng.

Self Cite

View full text Add to dashboard Cite

We present a simple, message-optimal algorithm for maintaining a random sample from a large data stream whose input elements are distributed across multiple sites that communicate via a central coordinator. At any point in time, the set of elements held by the coordinator represent a uniform random sample from the set of all the elements observed so far. When compared with prior work, our algorithms asymptotically improve the total number of messages sent in the system. We present a matching lower bound, showing that our protocol sends the optimal number of messages up to a constant factor with large probability. We also consider the important case when the distribution of elements across different sites is non-uniform, and show that for such inputs, our algorithm significantly outperforms prior solutions. Abstract-We present a simple, message-optimal algorithm for maintaining a random sample from a large data stream whose input elements are distributed across multiple sites that communicate via a central coordinator. At any point in time, the set of elements held by the coordinator represent a uniform random sample from the set of all the elements observed so far. When compared with prior work, our algorithms asymptotically improve the total number of messages sent in the system. We present a matching lower bound, showing that our protocol sends the optimal number of messages up to a constant factor with large probability. We also consider the important case when the distribution of elements across different sites is non-uniform, and show that for such inputs, our algorithm significantly outperforms prior solutions. Keywords

show abstract

Section: Related Workmentioning

confidence: 99%

A Simple Message-Optimal Algorithm for Random Sampling from a Distributed Stream

Chung

Tirthapura

Woodruff

2016

IEEE Trans. Knowl. Data Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…This challenge stems from the unbounded nature of data streams and the impossibility of storing them. We need to design single-pass [25] and low-time-complexity [31] algorithms analogously to what is done today in stream processing, where constant-time and polylog-space-complexity algorithms find a vast application [32]. Moreover, those models will need to explain why the predictions are correct and how much the users trust them.…”

Section: Challenges and Benefitsmentioning

confidence: 99%

“…Last, developing stateful incremental algorithms that learn one sample at a time is very important. The smaller the data to process, the less sophisticated and the more scalable the tools are [31].…”

Section: Ziffer Et Al / Towards Time-evolving Analytics 11mentioning

confidence: 99%

Towards time-evolving analytics: Online learning for time-dependent evolving data streams

Ziffer

Bernardo

Valle

et al. 2023

View full text Add to dashboard Cite

Traditional historical data analytics is at risk in a world where volatility, uncertainty, complexity, and ambiguity are the new normal. While Streaming Machine Learning (SML) and Time-series Analytics (TSA) attack some aspects of the problem, we still need a comprehensive solution. SML trains models using fewer data and in a continuous/adaptive way relaxing the assumption that data points are identically distributed. TSA considers temporal dependence among data points, but it assumes identical distribution. Every Data Scientist fights this battle with ad-hoc solutions. In this paper, we claim that, due to the temporal dependence on the data, the existing solutions do not represent robust solutions to efficiently and automatically keep models relevant even when changes occur, and real-time processing is a must. We propose a novel and solid scientific foundation for Time-Evolving Analytics from this perspective. Such a framework aims to develop the logical, methodological, and algorithmic foundations for fast, scalable, and resilient analytics.

show abstract

“…Namely, nearly all lower bounds for the space complexity of randomized streaming algorithms are derived via reductions from communication problems. For an incomplete list of such reductions, see [58,61,45,42,46,10,16,57,48,49,40] and the references therein. Now nearly all such lower bounds (and all of the ones that were just cited) hold in either the 2-party setting (G has 2 vertices), the coordinator model, or the black-board model.…”

Section: Multi-party Communicationmentioning

confidence: 99%

Towards Optimal Moment Estimation in Streaming and Distributed Models

Jayaram

Woodruff

2023

ACM Trans. Algorithms

Self Cite

View full text Add to dashboard Cite

One of the oldest problems in the data stream model is to approximate the p -th moment \(\Vert \mathbf {X}\Vert _p^p = \sum _{i=1}^n \mathbf {X}_i^p \) of an underlying non-negative vector \(\mathbf {X} \in \mathbb {R}^n \) , which is presented as a sequence of poly( n ) updates to its coordinates. Of particular interest is when p ∈ (0, 2]. Although a tight space bound of Θ (ϵ − 2 log n ) bits is known for this problem when both positive and negative updates are allowed, surprisingly there is still a gap in the space complexity of this problem when all updates are positive. Specifically, the upper bound is O (ϵ − 2 log n ) bits, while the lower bound is only Ω (ϵ − 2 + log n ) bits. Recently, an upper bound of \(\tilde{O}(\epsilon ^{-2} + \log n) \) bits was obtained under the assumption that the updates arrive in a random order . We show that for p ∈ (0, 1], the random order assumption is not needed. Namely, we give an upper bound for worst-case streams of \(\tilde{O}(\epsilon ^{-2} + \log n) \) bits for estimating \(\Vert \mathbf {X}\Vert _p^p \) . Our techniques also give new upper bounds for estimating the empirical entropy in a stream. On the other hand, we show that for p ∈ (1, 2], in the natural coordinator and blackboard distributed communication topologies, there is an \(\tilde{O}(\epsilon ^{-2}) \) bit max-communication upper bound based on a randomized rounding scheme. Our protocols also give rise to protocols for heavy hitters and approximate matrix product. We generalize our results to arbitrary communication topologies G , obtaining an \(\tilde{O}(\epsilon ^{2} \log d) \) max-communication upper bound, where d is the diameter of G . Interestingly, our upper bound rules out natural communication complexity-based approaches for proving an Ω (ϵ − 2 log n ) bit lower bound for p ∈ (1, 2] for streaming algorithms. In particular, any such lower bound must come from a topology with large diameter.

show abstract

Space-Efficient Estimation of Statistics Over Sub-Sampled Streams

Cited by 10 publications

References 35 publications

A Simple Message-Optimal Algorithm for Random Sampling from a Distributed Stream

A Simple Message-Optimal Algorithm for Random Sampling from a Distributed Stream

Towards time-evolving analytics: Online learning for time-dependent evolving data streams

Towards Optimal Moment Estimation in Streaming and Distributed Models

Contact Info

Product

Resources

About