Black-box determination of cost models' parameters for federated stream-processing systems

Abstract

For distribution and deployment of queries in distributed streamprocessing environments, it is vital to estimate the expected costs in advance. Having heterogeneous Stream-Processing Systems (SPSs) running on various hosts, the parameters of a cost model for an operator must be determined by measurements for each relevant combination of an SPS and hardware.This paper presents a black-box method that determines the parameters of appropriate cost models that regard system-specific behavior. For some SPSs, there might not be any appropriate cost model available due to the lack of internal knowledge. If no cost model is available for any reason, we provide and apply a nonparametric model.

Machines Tuning Machines: Configuring Distributed Stream Processors with Bayesian Optimization

“…Often, cost models have been proposed to capture complexities of these systems [4], [5] to optimize the use of resources [4], [6] or query execution [7], [8]. Others have applied Covariance Matrix Adaption (CMA) [9] or searched the parameter space using an experimentally constructed parameter dependency graph [1].…”

Data Stream Application Manager (DSAM)

“…For this decision, it uses a cost-based optimization that keeps the total resource consumption as low as possible while adhering to some additional constraints (e. g. not overloading any individual system). The execution costs of a possible distribution are estimated using either parametric or non-parametric cost models [1]. We currently use simulated annealing to choose a suitable distribution solution.…”