Feedback Autonomic Provisioning for Guaranteeing Performance in MapReduce Systems

Abstract

International audienceCompanies have a fast growing amounts of data to process and store, a data explosion is happening next to us. Currentlyone of the most common approaches to treat these vast data quantities are based on the MapReduce parallel programming paradigm.While its use is widespread in the industry, ensuring performance constraints, while at the same time minimizing costs, still providesconsiderable challenges. We propose a coarse grained control theoretical approach, based on techniques that have already provedtheir usefulness in the control community. We introduce the first algorithm to create dynamic models for Big Data MapReduce systems,running a concurrent workload. Furthermore we identify two important control use cases: relaxed performance - minimal resourceand strict performance. For the first case we develop two feedback control mechanism. A classical feedback controller and an evenbasedfeedback, that minimises the number of cluster reconfigurations as well. Moreover, to address strict performance requirements afeedforward predictive controller that efficiently suppresses the effects of large workload size variations is developed. All the controllersare validated online in a benchmark running in a real 60 node MapReduce cluster, using a data intensive Business Intelligenceworkload. Our experiments demonstrate the success of the control strategies employed in assuring service time constraints

A control-theory approach for cluster autonomic management: maximizing usage while avoiding overload

“…This can be done in response to monitored sensors of the systems, by analysis of this data and utilization of the results in order to trigger appropriate system-level or application-level reconfiguration mechanisms. There is not a single way to perform this reconfiguration, one of them being the development of feedback loops (see for example [1]) which in the domain of Computer Science are the object of Autonomic Computing [2].…”

A control theoretical view of cloud elasticity: taxonomy, survey and challenges

“…However, no such details are provided in the case of [72]. In contrast, the authors of [43,83] used a PI feedback controller for big data application. They focused to adjusts the computing nodes of a map reduce cluster to guarantee the desired service time of map reduce jobs.…”

An autonomic‐computing approach on mapping threads to multi‐cores for software transactional memory

“…Autonomic computing proposes a general structure of feedback loop to take adaptive and reconfigurable computing into account. [27][28][29] A classic feedback control loop is illustrated in Figure 2 in the shape of a MAPE-K (Monitor, Analyse, Plan, Execute, Knowledge) loop. [27][28][29] A classic feedback control loop is illustrated in Figure 2 in the shape of a MAPE-K (Monitor, Analyse, Plan, Execute, Knowledge) loop.…”

“…26 Feedback control loops, on one form or another, have been adopted as cornerstones of software-intensive and self-adaptive systems. [27][28][29] A classic feedback control loop is illustrated in Figure 2 in the shape of a MAPE-K (Monitor, Analyse, Plan, Execute, Knowledge) loop.…”