Learning with Analytical Models

Ibeid, Huda; Meng, Siping; Dobon, Oliver; Olson, Luke N.; Gropp, William

doi:10.1109/ipdpsw.2019.00128

Cited by 7 publications

(1 citation statement)

References 28 publications

(37 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To have a fair comparison between the two techniques, the optimum combination of the (itrT hr, stopT hr) thresholds was determined as in the previous section both for our hybrid and the IML approach. The optimum combination of the two thresholds are (38,24) A Hybrid Machine Learning Approach for Performance Modeling of Big Data Applications 11 and (30,19) for HML and IML approaches, respectively.…”

Section: Finding the Optimum Of The Thresholdsmentioning

confidence: 99%

A Hybrid Machine Learning Approach for Performance Modeling of Cloud-Based Big Data Applications

Ataie

Evangelinou

Gianniti

et al. 2021

The Computer Journal

View full text Add to dashboard Cite

Nowadays, Apache Hadoop and Apache Spark are two of the most prominent distributed solutions for processing big data applications on the market. Since in many cases these frameworks are adopted to support business critical activities, it is often important to predict with fair confidence the execution time of submitted applications, for instance when service-level agreements are established with end-users. In this work, we propose and validate a hybrid approach for the performance prediction of big data applications running on clouds, which exploits both analytical modeling and machine learning (ML) techniques and it is able to achieve a good accuracy without too many time consuming and costly experiments on a real setup. The experimental results show how the proposed approach attains improvement in accuracy, number of experiments to be run on the operational system and cost over applying ML techniques without any support from analytical models. Moreover, we compare our approach with Ernest, an ML-based technique proposed in the literature by the Spark inventors. Experiments show that Ernest can accurately estimate the performance in interpolating scenarios while it fails to predict the performance when configurations with increasing number of cores are considered. Finally, a comparison with a similar hybrid approach proposed in the literature demonstrates how our approach significantly reduce prediction errors especially when few experiments on the real system are performed.

show abstract