No abstract

Humanity entered the Big Data era passing through the zettabite mark at an exponential rate in just a few years, which made Big Data processing issues a reality for a wide range of business and academic applications. The Apache Hadoop is the most popular Big Data processing framework with the current market share of $3 billion expected to increase to $50 billion by the end of the decade; however, Hadoop deployment expansion does not proceed at the predicted rate.One of the obstacles for further popularization of Hadoop is that performance is highly dependent on chosen settings. A nontrivial tuning process requires time and highly qualified human resources. Therefore, an automatic tuner is a desirable solution. Existent off-line tuning approaches require multiple repetitive executions of the job in order to find optimal tuning settings and, hence, are not applicable to use in most cases.The presented work introduces a novel real time autotuning approach. The resource management parameters are tuned between execution waves by a modular autotuner connected to Hadoop architecture through YARN. The developed autotuner effectively intercepts a resource request, modifies it according to a tuning algorithm and passes it to YARN Scheduler. Such an approach not only carries high practical and theoretical value, but also opens a new horizon in the Hadoop/YARN automatic tuning development.iii This thesis is dedicated to my husband, Pino Guerra, and to my children, Matteo and Isabella. Thank you for your love, patience, and numerous sacrifices that you have made throughout my academic career. This thesis and the pursuit of my goals would not have been possible without your support.iv