What's hot and what's not: tracking most frequent items dynamically

Abstract

Most database management systems maintain statistics on the underlying relation. One of the important statistics is that of the "hot items" in the relation: those that appear many times (most frequently, or more than some threshold). For example, end-biased histograms keep the hot items as part of the histogram and are used in selectivity estimation. Hot items are used as simple outliers in data mining, and in anomaly detection in networking applications.We present a new algorithm for dynamically determining the hot items at any time in the relation that is undergoing deletion operations as well as inserts. Our algorithm maintains a small space data structure that monitors the transactions on the relation, and when required, quickly outputs all hot items, without rescanning the relation in the database. With user-specified probability, it is able to report all hot items. Our algorithm relies on the idea of "group testing", is simple to implement, and has provable quality, space and time guarantees. Previously known algorithms for this problem that make similar quality and performance guarantees can not handle deletions, and those that handle deletions can not make similar guarantees without rescanning the database. Our experiments with real and synthetic data shows that our algorithm is remarkably accurate in dynamically tracking the hot items independent of the rate of insertions and deletions.

DBToaster: higher-order delta processing for dynamic, frequently fresh views

“…load shedding, prioritization [30]), on shared processing (e.g. on-the-fly aggregation [21]), or specialized algorithms and data structures [11]. Our approach to streaming is about generalizing incremental processing to (nonwindowed) SQL semantics (including nested subqueries and aggregates).…”

Fading histograms in detecting distribution and concept changes

“…The online update of such structures in a dynamic scenario is also a required property. Sampling [39], hot lists [11,30], wavelets [9,18,24], sketches [10] and histograms [21][22][23] are examples of synopses methods to obtain fast and approximated answers.…”

Data Streams: Algorithms and Applications

“…Mainly, they (and the techniques) are used as subroutine in other problems in the Turnstile data stream model. F 2 and L 2 are used to measure deviations in anomaly detection [154] or interpreted as self-join sizes [13]; with variants of L 1 sketches we can dynamically track most frequent items [55], quantiles [107], wavelets and histograms [104], etc. in the Turnstile model; using L p sketches for p → 0, we can estimate the number of distinct elements at any time in the Turnstile model [47].…”