Disseminating streaming data in a dynamic environment: an adaptive and cost-based approach

Abstract

In a distributed stream processing system, streaming data are continuously disseminated from the sources to the distributed processing servers. To enhance the dissemination efficiency, these servers are typically organized into one or more dissemination trees. In this paper, we focus on the problem of constructing dissemination trees to minimize the average loss of fidelity of the system. We observe that existing heuristic-based approaches can only explore a limited solution space and hence may lead to sub-optimal solutions. On the contrary, we propose an adaptive and cost-based approach. Our cost model takes into account both the processing cost and the communication cost. Furthermore, as a distributed stream processing system is vulnerable to inaccurate statistics, runtime fluctuations of data characteristics, server workloads, and network conditions, we have designed our scheme to be adaptive to these situations: an operational dissemination tree may be incrementally transformed to a more cost-effective one. Our adaptive strategy employs distributed decisions made by the distributed servers independently based on localized statistics collected by each server at runtime. For a relatively static environment, we also propose two static tree construction algorithms relying on apriori system statistics. These static trees can also be used as initial trees in a dynamic environment. We apply our schemes to both single-and multi-object dissemination. Our extensive performance study shows that the adaptive mechanisms are effective in a dynamic context and the proposed static tree construction algorithms perform close to optimal in a static environment.

Efficiently Maintaining Distributed Model-Based Views on Real-Time Data Streams

“…After the initialization, the producer node represents the stream as a model instance M. 1 during t ∈ [1,4], and it simultaneously matches the actual parameters for M. 1 to the most similar preset parameter values (i.e., 0.7 and 1 for slope and v-intercept, respectively). While this, the producer node also monitors whether Property 1 holds between each actual sensor reading and its corresponding model-driven value obtained from the preset model.…”

“…While this, the producer node also monitors whether Property 1 holds between each actual sensor reading and its corresponding model-driven value obtained from the preset model. At the consumer node, the model-based view is generated by the predetermined model during the same time period t ∈ [1,4].…”

“…Streaming data dissemination [3], [4] concerns continuous data transfers from producer nodes to consumer nodes. These studies assume that raw data streams must be disseminated.…”

“…A rich body of research deals with this problem, including communication-efficient data dissemination [3], [4] and effective operator placement in distributed stream processing systems [5], [6], [7]. Although these proposals reduce large amounts of data transfers over distributed sensor networks, they are applicable only for particular query types, or inefficient when a query result contains a large volume of data.…”

On processing continuous frequent K-N-match queries for dynamic data over networked data sources

“…In applications such as stock markets, moving objects, sensor networks, and intrusion detection systems, a large volume of data are stored in databases and the values of these data usually change frequently [27,37]. Thus, the result of a query may change as the attributes of objects change.…”