Scuba

Abraham, Lior; Allen, J. L.; Barykin, Oleksandr; Borkar, Vinayak; Chopra, Bhuwan; Gerea, Ciprian; Merl, Daniel; Metzler, Josh; Reiss, David J; Subramanian, S.; Wiener, Joshua; Zed, Okay

doi:10.14778/2536222.2536231

Cited by 55 publications

(14 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…During moments sketch quantile estimation we run Newton's method until the moments match to within δ = 10 −9 , and select k 1 , k 2 using a maximum condition number κ max = 10 4 . We construct the moments sketch to store both standard and log moments 2 up to order k, but choose at query time which moments to make use of as described in Section 4.3.2. We quantify the accuracy of a quantile estimate using the quantile error ε as defined in Section 3.1.…”

Section: Methodsmentioning

confidence: 99%

“…High-performance aggregation. The aggregation scenarios in Section 1 are found in many existing streaming data systems [8,16,24,65,82], as well as data cube [33,69], data exploration [2], and visualization [17] systems. In particular, these systems are can perform interactive aggregations over time windows and along many cube dimensions, motivating the design of our sketch.…”

Section: Related Workmentioning

confidence: 99%

“…Performing interactive multi-dimensional analytics over data from sensors, devices, and servers increasingly requires computing aggregate statistics for specific subpopulations and time windows [4,29,65]. In applications such as A/B testing [38,42], exploratory data analysis [8,74], and operations monitoring [2,14], analysts perform aggregation queries to understand how specific user cohorts, device types, and feature flags are behaving. In particular, computing quantiles over these subpopulations is an essential part of debugging and real-time monitoring workflows [26].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Moment-based quantile sketches for efficient high cardinality aggregation queries

Gan¹,

Ding²,

Tai³

et al. 2018

Proc. VLDB Endow.

View full text Add to dashboard Cite

Interactive analytics increasingly involves querying for quantiles over sub-populations of high cardinality datasets. Data processing engines such as Druid and Spark use mergeable summaries to estimate quantiles, but summary merge times can be a bottleneck during aggregation. We show how a compact and efficiently mergeable quantile sketch can support aggregation workloads. This data structure, which we refer to as the moments sketch, operates with a small memory footprint (200 bytes) and computationally efficient (50ns) merges by tracking only a set of summary statistics, notably the sample moments. We demonstrate how we can efficiently estimate quantiles using the method of moments and the maximum entropy principle, and show how the use of a cascade further improves query time for threshold predicates. Empirical evaluation shows that the moments sketch can achieve less than 1 percent quantile error with 15× less overhead than comparable summaries, improving end query time in the MacroBase engine by up to 7× and the Druid engine by up to 60×. *

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Moment-based quantile sketches for efficient high cardinality aggregation queries

Gan¹,

Ding²,

Tai³

et al. 2018

Proc. VLDB Endow.

View full text Add to dashboard Cite

show abstract

“…Taggers then convert each annotated packet header into a JSON object and feed it into Scuba [3], a real-time data analytics system. Samples are simultaneously stored into Hive [38] tables for long-term analysis.…”

Section: Fbflowmentioning

confidence: 99%

Inside the Social Network's (Datacenter) Network

Roy

Zeng

Bagga

et al. 2015

SIGCOMM Comput. Commun. Rev.

230

176

View full text Add to dashboard Cite

Large cloud service providers have invested in increasingly larger datacenters to house the computing infrastructure required to support their services. Accordingly, researchers and industry practitioners alike have focused a great deal of effort designing network fabrics to efficiently interconnect and manage the traffic within these datacenters in performant yet efficient fashions. Unfortunately, datacenter operators are generally reticent to share the actual requirements of their applications, making it challenging to evaluate the practicality of any particular design.Moreover, the limited large-scale workload information available in the literature has, for better or worse, heretofore largely been provided by a single datacenter operator whose use cases may not be widespread. In this work, we report upon the network traffic observed in some of Facebook's datacenters. While Facebook operates a number of traditional datacenter services like Hadoop, its core Web service and supporting cache infrastructure exhibit a number of behaviors that contrast with those reported in the literature. We report on the contrasting locality, stability, and predictability of network traffic in Facebook's datacenters, and comment on their implications for network architecture, traffic engineering, and switch design.

show abstract

“…The streaming join service was implemented in the context of two of Facebook's stream processing platforms: Puma and Stylus. Both systems ingest data from Scribe [15] -a persistent message bus -and can later publish data back to Scribe, Scuba [8], or Hive.…”

Section: Systems Overviewmentioning

confidence: 99%

Providing streaming joins as a service at Facebook

et al. 2018

View full text Add to dashboard Cite

Stream processing applications reduce the latency of batch data pipelines and enable engineers to quickly identify production issues. Many times, a service can log data to distinct streams, even if they relate to the same real-world event (e.g., a search on Facebook's search bar). Furthermore, the logging of related events can appear on the server side with different delay, causing one stream to be significantly behind the other in terms of logged event times for a given log entry. To be able to stitch this information together with low latency, we need to be able to join two different streams where each stream may have its own characteristics regarding the degree in which its data is out-of-order. Doing so in a streaming fashion is challenging as a join operator consumes lots of memory, especially with significant data volumes. This paper describes an end-to-end streaming join service that addresses the challenges above through a streaming join operator that uses an adaptive stream synchronization algorithm that is able to handle the different distributions we observe in real-world streams regarding their event times. This synchronization scheme paces the parsing of new data and reduces overall operator memory footprint while still providing high accuracy. We have integrated this into a streaming SQL system and have successfully reduced the latency of several batch pipelines using this approach. PVLDB Reference Format:

show abstract

Scuba

Cited by 55 publications

References 5 publications

Moment-based quantile sketches for efficient high cardinality aggregation queries

Moment-based quantile sketches for efficient high cardinality aggregation queries

Inside the Social Network's (Datacenter) Network

Providing streaming joins as a service at Facebook

Contact Info

Product

Resources

About