Improving sketch reconstruction accuracy using linear least squares method

Lee, Gene Moo; Liu, Huiya; Yoon, Young; Zhang, Yin

doi:10.1145/1330107.1330138

Cited by 17 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…When the added items are heavy hitters, this can substantially reduce the magnitude of the error. The choice of regression model is thus dictated by what one knows about the universe of items and the unknown error distribution Under the assumption that the error distribution is normal and only a subset S of items are known, one recovers the linear least squares method of [19]. This is equivalent to the solution of the maximization problem…”

Section: Regression and Support Constraintsmentioning

confidence: 99%

“…For example, when there are few heavy hitters and a large number of items, the Count-Min sketch can be highly biased and perform poorly compared to the Count sketch [3]. This has led to a number of attempts [16], [19], [10], [20], [4] to improve estimation from the Count-Min sketch in these regimes. In all cases, these methods can be shown to perform suboptimally in some regimes or for some sketch parameter settings and often worse than the basic Count-Min estimator.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Count-Min

Ting

2018

Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

The Count-Min sketch is an important and well-studied data summarization method. It can estimate the count of any item in a stream using a small, xed size data sketch. However, the accuracy of the Count-Min sketch depends on characteristics of the underlying data. This has led to a number of count estimation procedures which work well in one scenario but perform poorly in others. A practitioner is faced with two basic, unanswered questions. Given an estimate, what is its error? Which estimation procedure should be chosen when the data is unknown? We provide answers to these questions. We derive new count estimators, including a provably optimal estimator, which best or match previous estimators in all scenarios. We also provide practical, tight error bounds at query time for all estimators and methods to tune sketch parameters using these bounds. The key observation is that the full distribution of errors in each counter can be empirically estimated from the sketch itself. By rst estimating this distribution, count estimation becomes a statistical estimation and inference problem with a known error distribution. This provides both a principled way to derive new and optimal estimators as well as a way to study the error and properties of existing estimators.

show abstract

Section: Regression and Support Constraintsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Count-Min

Ting

2018

Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

show abstract

“…In addition to the countmin estimator, a leastsquare estimator of the heavy hitters was proposed in [9] for the linear regression problem in (9), which can be viewed as the maximum likelihood estimate when the error distribution follows a normal distribution. In Section VI, we shall compare with least square method and show that the estimates obtained from solving the linear programming problem are superior.…”

Section: A Heavy Hitter Estimationmentioning

confidence: 99%

“…In terms of the accuracy of estimation, we show that linear regression provides a better estimate of values of heavy hitters than does the least-square method proposed in [9]. Here, we consider the following two error measures:…”

Section: Experiments 2 (Analysis Of Finding Heavy Hitters (With Linearmentioning

confidence: 99%

A Fast and Compact Method for Unveiling Significant Patterns in High Speed Networks

Cao

Chen

et al. 2007

IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications

View full text Add to dashboard Cite

Abstract-Identification of significant patterns in network traffic, such as IPs or flows that contribute large volume (heavy hitters) or introduce large changes (heavy changers), has many applications in accounting and network anomaly detection. As network speed and the number of flows grow rapidly, tracking per-IP or per-flow statistics becomes infeasible due to both the computational overhead and memory requirements. In this paper, we propose a novel sequential hashing scheme that requires only O(H log N ) both in memory and computational overhead that are close to being optimal, where N is the the number of all possible keys (e.g., flows, IPs) and H is the maximum number of heavy keys. Moreover, the generalized sequential hashing scheme makes it possible to trade off among memory, update cost, and detection cost in a large range that can be utilized by different computer architectures for optimizing the overall performance. In addition, we also propose statistically efficient algorithms for estimating the values of heavy hitters and heavy changers. Using both theoretical analysis and experimental studies of Internet traces, we demonstrate that our approach can achieve the same accuracy as the existing methods do but using much less memory and computational overhead.

show abstract

“…For example, let be a i = (j, I i ), I i > 0, and A[j] i = A i−1 + I i . In this case, A is a ow in the cash register model, where A[j] i is the signal state after the arrival of the i-th sample[58]. This model is one of the most popular.…”

mentioning

confidence: 99%

A Survey on Data Analysis on Large-Scale Wireless Networks: Online Stream Processing, Trends, and Challenges

Medeiros

Neto

Lopez

et al. 2020

Preprint

View full text Add to dashboard Cite

In this paper we focus on knowledge extraction from large-scale wireless networks through stream processing. We present the primary methods for sampling, data collection, and monitoring of wireless networks and we characterize knowledge extraction as a machine learning problem on big data stream processing. We show the main trends in big data stream processing frameworks. Additionally, we explore the data preprocessing, feature engineering, and the machine learning algorithms applied to the scenario of wireless network analytics. We address challenges and present research projects in wireless network monitoring and stream processing. Finally, future perspectives, such as deep learning and reinforcement learning in stream processing, are anticipated.

show abstract

Improving sketch reconstruction accuracy using linear least squares method

Cited by 17 publications

References 0 publications

Count-Min

Count-Min

A Fast and Compact Method for Unveiling Significant Patterns in High Speed Networks

A Survey on Data Analysis on Large-Scale Wireless Networks: Online Stream Processing, Trends, and Challenges

Contact Info

Product

Resources

About