Efficient and cost-aware operator placement in heterogeneous stream-processing environments

Daum, Michael; Lauterwald, Frank; Baumgärtel, Philipp; Pollner, Niko; Meyer-Wegener, Klaus

doi:10.1145/2002259.2002327

Cited by 5 publications

(4 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the initial placement problem we use the TAP model from [2]. We have implemented five different heuristic algorithms: Simulated annealing [7], standard particle swarm [3], ranked fully informed particle swarm [6], hybrid particle swarm [12] and orthogonal particle swarm [11].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Operator fission for load balancing in distributed heterogeneous data stream processing systems

Pollner

Steudtner

Meyer-Wegener

2015

Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems

Self Cite

View full text Add to dashboard Cite

Distributed data stream processing allows to optimize resource consumption. A query's operators can be executed by several systems. The placement of filter or aggregate operators near the data source omits unnecessary data transfer. The operator placement decision is a complex problem. In certain scenarios the goal is not only overall minimization of e.g. resource consumption but an evenly distributed load.We propose an operator fission algorithm, that works on the basis of an initial operator placement. The algorithm selects certain operators from the set of operators that allow fission for parallel execution by multiple systems. Load is thus divided between processors in a more fine-grained way, resulting in lower maximum load and lower load variance. We present and evaluate three different variants of the algorithm to allow tuning the trade-off between optimization time and result quality.

show abstract

Section: Discussionmentioning

confidence: 99%

“…There exist several heuristic algorithms that can provide such a placement, e.g. those in [2]. We don't assume any special algorithm and we don't assume that it optimizes for load balancing.…”

Section: Operator Fissionmentioning

confidence: 99%

Operator fission for load balancing in distributed heterogeneous data stream processing systems

Pollner

Steudtner

Meyer-Wegener

2015

Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…Daum et al proposed a framework called Data Stream Application Manager (DSAM) Lauterwald et al (2012) to control a network of heterogeneous data stream management systems. A cost model Daum et al (2011) is used to minimize the overall processing and communication costs where the operator placement query optimization is modelled as a task assignment problem.…”

Section: Related Workmentioning

confidence: 99%

Data-sharing markets for integrating IoT data processing functionalities

Kasrin

Benabbas

Elmamooz

et al. 2021

CCF Trans. Pervasive Comp. Interact.

View full text Add to dashboard Cite

The recent evolution of the Internet of Things into a cyber-physical reality has spawned various challenges from a data management perspective. In addition, IoT platform designers are faced with another set of questions. How can platforms be extended to smoothly integrate new data management functionalities? Currently, data processing related tasks are typically realized by manually developed code and functions which creates difficulties in maintenance and growth. Hence we need to explore other approaches to integration for IoT platforms. In this paper we cover both these aspects: (1) we explore several emerging data management challenges, and (2) we propose an IoT platform integration model that can combine disparate functionalities under one roof. For the first, we focus on the following challenges: sensor data quality, privacy in data streams, machine learning model management, and resource-aware data management. For the second, we propose an information-integration model for IoT platforms. The model revolves around the concept of a Data-Sharing Market where data management functionalities can share and exchange information about their data with other functionalities. In addition, data-sharing markets themselves can be combined into networks of markets where information flows from one market to another, which creates a web of information exchange about data resources. To motivate this work we present a use-case application in smart cities.

show abstract

“…We currently use simulated annealing to choose a suitable distribution solution. However, we already evaluated several alternative algorithms [2] with even more algorithms currently under evaluation. The distribution process yields so-called partial queries that each represent the part of a query that should be executed on one target DSMS.…”

Section: Dsam Architecturementioning

confidence: 99%

Data Stream Application Manager (DSAM)

Lauterwald

Pollner

Daum

et al. 2012

Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems

Self Cite

View full text Add to dashboard Cite

We present our Data Stream Application Manager (DSAM). DSAM integrates different data stream management systems (DSMS) and also sensor network nodes. The user enters a query written in a system-independent declarative query language into a central control program. DSAM analyzes this query and distributes it to the available DSMSs. The distribution algorithm uses a cost model to minimize overall processing and transmission costs. The demo shows some scenarios in which DSAM can be used. We do not focus on concrete applications but on generic use-cases where DSAM may help users or application developers. One example is switching to a different DSMS without changing queries. Another example is the automatic integration of different DSMSs to make them appear just like one system.

show abstract

Efficient and cost-aware operator placement in heterogeneous stream-processing environments

Cited by 5 publications

References 2 publications

Operator fission for load balancing in distributed heterogeneous data stream processing systems

Operator fission for load balancing in distributed heterogeneous data stream processing systems

Data-sharing markets for integrating IoT data processing functionalities

Data Stream Application Manager (DSAM)

Contact Info

Product

Resources

About