Towards automatic parameter tuning of stream processing systems

Bilal, Muhammad; Canini, Marco

doi:10.1145/3127479.3127492

Cited by 43 publications

(33 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For grey-box systems, [20] uses system-level monitoring information in conjunction with standard hill climbing algorithms to significantly improve MapReduce application performance. [4] demonstrate benefits of using automated parameter tuning in optimizing big-data streaming applications. In their work, they transform their multi-objective optimization function into a single-objective optimization problem and show that their rule-based approach of incorporating prior knowledge for parameter selection allows them to converge significantly faster than standard hill-climbing algorithms used for typical black box problems.…”

Section: Related Workmentioning

confidence: 99%

“…There has been significant recent work in how parameter tuning can be done better by leveraging knowledge of the internal implementation [4,16,20]. If we have full knowledge of the internals of a search system, we would have a white-box system.…”

Section: Related Workmentioning

confidence: 99%

“…In this case, even though the parameter values and output scores for the system are known, the validation set Q is near-trivial as it only contains s f as no internal subscores s i are logged, and as such no offline evaluation at all can be done. 4 However, often times we may have some knowledge of the internals of a system [4,20]. An example of this is a search system where the general graph of computation is known, as in [16], but different scoring functions f i and subscores s i are unknown (type(f i ) = type(s i ) = hidden).…”

Section: Black-box and Grey-box Formulationmentioning

confidence: 99%

“…For instance, we may have no idea how the query parsing library works. Or, we may have exact knowledge of how a BM25 score is produced but do not actually have the 4 In this case, a common solution to doing performance tuning is to either do black-box tuning or build a reranking system on top and leave the complex system untouched. logging capability to retrieve that subscore.…”

Section: Black-box and Grey-box Formulationmentioning

confidence: 99%

See 3 more Smart Citations

Parameter Tuning in Personal Search Systems

Chen

Wang

Qin

et al. 2020

Proceedings of the 13th International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

The effectiveness of information retrieval systems is heavily dependent on how various parameters are tuned. One option to find these parameters is to run multiple online experiments using a parameter sweep approach in order to optimize the search system. There are multiple downsides of this approach, including the fact that it may lead to a poor experience for users. Another option is to do offline evaluation, which can act as a safeguard against potential quality issues. Offline evaluation requires a validation set of data that can be benchmarked against different parameter settings. However, for search over personal corpora, e.g. email and file search, it is impractical and often impossible to get a complete representative validation set due to the inability to save raw queries and document information. In this work, we show how to do offline parameter tuning with only a partial validation set. In addition, we demonstrate how to do parameter tuning in situations when we have complete knowledge of the internal implementation of the search system (white-box tuning), as well as situations where we have only partial knowledge (grey-box tuning). The resulting method provides a way of performing offline parameter tuning in a privacy-preserving manner. We demonstrate the effectiveness of the proposed approach by reporting the results from search ranking experiments performed on two large-scale personal search systems. CCS CONCEPTS• Information systems → Evaluation of retrieval results.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Black-box and Grey-box Formulationmentioning

confidence: 99%

Section: Black-box and Grey-box Formulationmentioning

confidence: 99%

See 2 more Smart Citations

Parameter Tuning in Personal Search Systems

Chen

Wang

Qin

et al. 2020

Proceedings of the 13th International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

show abstract

“…Clipper adopts a containerized design allowing each model to be individually managed, configured, and deployed in separate containers, but does not support prediction pipelines or reasoning about latency deadlines across models. Several systems have explored offline pipeline configuration for data pipelines [6,16]. However, these target generic data streaming pipelines.…”

Section: Related Workmentioning

confidence: 99%

InferLine

Crankshaw

Sela

et al. 2020

Proceedings of the 11th ACM Symposium on Cloud Computing

View full text Add to dashboard Cite

Serving ML prediction pipelines spanning multiple models and hardware accelerators is a key challenge in production machine learning. Optimally configuring these pipelines to meet tight end-to-end latency goals is complicated by the interaction between model batch size, the choice of hardware accelerator, and variation in the query arrival process. In this paper we introduce InferLine, a system which provisions and manages the individual stages of prediction pipelines to meet end-to-end tail latency constraints while minimizing cost. InferLine consists of a low-frequency combinatorial planner and a high-frequency auto-scaling tuner. The low-frequency planner leverages stage-wise profiling, discrete event simulation, and constrained combinatorial search to automatically select hardware type, replication, and batching parameters for each stage in the pipeline. The high-frequency tuner uses network calculus to auto-scale each stage to meet tail latency goals in response to changes in the query arrival process. We demonstrate that InferLine outperforms existing approaches by up to 7.6x in cost while achieving up to 34.5x lower latency SLO miss rate on realistic workloads and generalizes across state-of-the-art model serving frameworks.

show abstract