Continuously adaptive continuous queries over streams

Madden, Samuel; Shah, Mehul A.; Hellerstein, Joseph M.; Raman, V.

doi:10.1145/564691.564698

Cited by 367 publications

(207 citation statements)

References 16 publications

Supporting

Mentioning

200

Contrasting

Unclassified

Order By: Relevance

“…Since the dissemination system runs continuously, it can experience these changes at runtime, which would make the previously optimal scheme sub-optimal. The problem of adapting to inaccurate statistics and system changes has been extensively explored in other problems such as query processing [2,15]. Unfortunately, few efforts have been devoted to adapting the structure of a dissemination tree at runtime.…”

Section: Motivationmentioning

confidence: 99%

Disseminating streaming data in a dynamic environment: an adaptive and cost-based approach

2007

View full text Add to dashboard Cite

In a distributed stream processing system, streaming data are continuously disseminated from the sources to the distributed processing servers. To enhance the dissemination efficiency, these servers are typically organized into one or more dissemination trees. In this paper, we focus on the problem of constructing dissemination trees to minimize the average loss of fidelity of the system. We observe that existing heuristic-based approaches can only explore a limited solution space and hence may lead to sub-optimal solutions. On the contrary, we propose an adaptive and cost-based approach. Our cost model takes into account both the processing cost and the communication cost. Furthermore, as a distributed stream processing system is vulnerable to inaccurate statistics, runtime fluctuations of data characteristics, server workloads, and network conditions, we have designed our scheme to be adaptive to these situations: an operational dissemination tree may be incrementally transformed to a more cost-effective one. Our adaptive strategy employs distributed decisions made by the distributed servers independently based on localized statistics collected by each server at runtime. For a relatively static environment, we also propose two static tree construction algorithms relying on apriori system statistics. These static trees can also be used as initial trees in a dynamic environment. We apply our schemes to both single-and multi-object dissemination. Our extensive performance study shows that the adaptive mechanisms are effective in a dynamic context and the proposed static tree construction algorithms perform close to optimal in a static environment.

show abstract

Section: Motivationmentioning

confidence: 99%

Disseminating streaming data in a dynamic environment: an adaptive and cost-based approach

2007

View full text Add to dashboard Cite

show abstract

“…They should be evaluated in real-time, which requires strict time and space constraints. Since a number of continuous queries are registered together in advance, it is more efficient to evaluate multiple queries collectively by sharing the common constraints of the queries, as already proposed in most DSMS's (Chandrasekaran and Franklin 2002;Chen et al 2000;Madden et al 2002;Sharaf et al 2007).…”

mentioning

confidence: 99%

Adaptive two-level optimization for selection predicates of multiple continuous queries

Lee

2012

J Intell Inf Syst

View full text Add to dashboard Cite

A data stream is a massive unbounded sequence of data elements continuously generated at a rapid rate. Query processing for such a data stream should also be continuous and rapid, which requires strict time and space constraints. In order to guarantee these constraints, we have proposed a new scheme called an Attribute Selection Construct (ASC) for an attribute of a data stream in our previous study (Lee and Lee, Information Sciences 178:2416-2432, 2008. As its optimization technique, this paper proposes the new strategy that determines the evaluation order of multiple ASC's for a given query set at two different levels-macro and micro levels. Based on the two levels, it also proposes two different strategies-macrosequence and hybrid-sequence-that find the optimized full evaluation sequence of all the ASC's. In addition, it provides the adaptive strategy that periodically rearranges the evaluation sequence of multiple ASC's. The performance of the proposed technique is verified by a series of experiments.

show abstract

“…Continuous query systems like [16,7,4] are essentially a collection of pipelined dataflows, a generalization of pipelined query plans, that are shared across numerous queries. A dataflow pipeline is a DAG in which the nodes are non-blocking operators, and the edges represent the direction in which data is processed.…”

Section: The Problem Of Changing Loadsmentioning

confidence: 99%

“…NiagraCQ [7] and XFilter [2] are examples of CQ systems over streaming XML documents. CACQ [16] is an adaptive system for relational-style continuous queries over streaming sensor data. PSoup [6] extends CACQ to accommodate disconnected operation.…”

Section: Related Workmentioning

confidence: 99%

“…Previous approaches have addressed these scalability issues in two ways. One thrust has been to leverage the commonality amongst queries [6,16,7,2] and index queries like data in a traditional database system. Another direction has been to exploit weaker semantics and sacrifice result quality for cases in which the system cannot be scaled to match peak workloads [18,4].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation