On Modelling and Prediction of Total CPU Usage for Applications in MapReduce Environments

Rizvandi, Nikzad Babaii; Taheri, Javid; Moraveji, Reza; Zomaya, Albert Y.

doi:10.1007/978-3-642-33078-0_30

Cited by 19 publications

(15 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The number of servers per PN varied between 10 and 60 for all evaluated scenarios. Note that we used processor cycles in GHz as a measure of the total processing capability of a node [30]. Table II summarizes the input parameters to the model.…”

Section: A Results Of Veracity Scenariosmentioning

confidence: 99%

Greening big data networks: velocity impact

Al-Salim

El-Gorashi

Lawey

et al. 2018

IET Optoelectronics

View full text Add to dashboard Cite

I IP WDM Wprocessing velocity of big data into two modes: expedited-data processing mode and relaxed-data processing mode. Expedited-data demands higher amount of computational resources to reduce the execution time compared to the relaxed-W M I L P MILP modes at strategic locations, dubbed processing nodes (PNs), built into the network along the path from the data source to the destination. During the processing of big data, the extracted information from the raw traffic is smaller in volume compared to the original big data traffic each time the data is processed, hence, reducing network power consumption. Our results showed that up to 60% network power saving is achieved when nearly 100% of the data in the network required relaxed-processing. In contrast, only 15% of network power saving is gained when nearly 100% of the data required expedited-processing. We obtained around 33% power saving in the mixed modes (i.e., when approximately 50% of the data is processed in the relaxed-mode and 50% of the data is processed in expedited-mode), compared to the classical approach where no PNs exist in the network and all the processing is achieved inside the centralized datacenters only. IntroductionVelocity is data in motion, which is the speed at which data is fluxing in and processed in the data centers [1]. The flux rate can grow larger for applications collecting information from wide spatial or temporal domains. For instance, the Square Kilometre Array [2] telescope combines signals with a flow speed of 700TB/second of data received from thousands of small antennas spread over a distance of more than 3000 km. In another example, five million trade events created each day are scrutinized in real time to identify potential fraud. Five hundred million daily call detail records are analysed in real-time to predict customer churn faster [3].High-speed processing of such immense data volumes as produced by plentiful data sources calls for new processing and communications methodologies in the big data era. In [4] the authors study the minimization of overall cost for Big Data placement, processing, and movement across geodistributed datacenters. In [5], the authors presented an optimization technique to execute a sequence of MapReduce jobs in Geo-distributed DCs to minimize the time and pecuniary cost. The authors in [6] introduced technique to execute MapReduce jobs on multiple IoT nodes to locally process as much data as possible the raw data. The authors in [7] aimed to minimize the communication cost by satisfying as many big data queries as possible over a number of time slots. In-network processing is proposed in [8] to achieve network-awareness to save more bandwidth using custom routing, redundancy elimination and on-path data reduction. In [9], the authors developed a Mixed Integer Linear programing models for energy efficient cloud computing services in IP over WDM core networks.We developed in [10] and [11] MILP models to investigate the impact of the big data's volume, variety, and veracity on gre...

show abstract

Section: A Results Of Veracity Scenariosmentioning

confidence: 99%

Greening big data networks: velocity impact

Al-Salim

El-Gorashi

Lawey

et al. 2018

IET Optoelectronics

View full text Add to dashboard Cite

show abstract

“…Also, Rahman et al [56] and Tiwari et al [65] studied the effect of compiler parameters on both performance and power/energy consumption for scientific computing. A lot of modeling and tuning effort has recently been devoted to the specific application of MapReduce [5,15,30,49,57,58].…”

Section: Related Workmentioning

confidence: 99%

Automated design of self-adaptive software with control-theoretical formal guarantees

Filieri

Hoffmann

Maggio

2014

Proceedings of the 36th International Conference on Software Engineering

123

View full text Add to dashboard Cite

Self-adaptation enables software to execute successfully in dynamic, unpredictable, and uncertain environments.Control theory provides a broad set of mathematically grounded techniques for adapting the behavior of dynamic systems. While it has been applied to specific software control problems, it has proved difficult to define methodologies allowing non-experts to systematically apply control techniques to create adaptive software. These difficulties arise because computer systems are usually non-linear, with varying workloads and heterogeneous components, making it difficult to model software as a dynamic system; i.e., by means of differential or difference equations. This paper proposes a broad scope methodology for automatically constructing both an approximate dynamic model of a software system and a suitable controller for managing its non-functional requirements. Despite its generality, this methodology provides formal guarantees concerning the system's dynamic behavior by keeping its model continuously updated to compensate for changes in the execution environment and effects of the initial approximation.We apply the methodology to three case studies, demonstrating its generality by tackling different domains (and different non-functional requirements) with the same approach. Being broadly applicable and fully automated, this methodology may allow the adoption of control theoretical solutions (and their formal properties) for a wide range of software adaptation problems.

show abstract

“…6) Physical link capacity constraints (27) Constraint (27) ensures that the summation of the wavelengths in a virtual link traversing a physical link does not exceed the capacity of the fibers in the physical link. 7) Wavelengths capacity constraint (28) Constraint (28) ensures that the summation of the wavelengths traversing a physical link does not exceed the total number of wavelengths in that link. 8) Number of aggregation ports utilized by regular traffic constraint (29) 9) Number of aggregation ports utilized by CHT traffic constraint (30) 10) Number of aggregation ports utilized by INF traffic constraint (31) Constraints (29)(30)(31) insight into what minimizes power consumption in our proposed progressive processing big data networks.…”

Section: Power Consumption Of Optical Switch Installed At Node I N (Wmentioning

confidence: 99%

“…The storage capacity of the PNs was assigned to be large enough. Note that we used processor cycles in GHz as a measure of the total processing capability of a node [28]. Table III summarizes the input parameters.…”

Section: Complexity Analysismentioning

confidence: 99%

Energy Efficient Big Data Networks: Impact of Volume and Variety

Al-Salim

Lawey

El-Gorashi

et al. 2018

IEEE Trans. Netw. Serv. Manage.

View full text Add to dashboard Cite

Abstract-In this article, we study the impact of big data's volume and variety dimensions on Energy Efficient Big Data Networks (EEBDN) by developing a Mixed Integer Linear Programming (MILP) model to encapsulate the distinctive features of these two dimensions. Firstly, a progressive energy efficient edge, intermediate, and central processing technique is proposed to process big data's raw traffic by building processing nodes (PNs) in the network along the way from the sources to datacenters. Secondly, we validate the MILP operation by developing a heuristic that mimics, in real time, the behaviour of the MILP for the volume dimension. Thirdly, we test the energy efficiency limits of our green approach under several conditions where PNs are less energy efficient in terms of processing and communication compared to data centers. Fourthly, we test the performance limits in our energy efficient approach by studying a "software matching" problem where different software packages are required to process big data. The results are then compared to the Classical Big Data Networks (CBDN) approach where big data is only processed inside centralized data centers. Our results revealed that up to 52% and 47% power saving can be achieved by the EEBDN approach compared to the CBDN approach, under the impact of volume and variety scenarios, respectively. Moreover, our results identify the limits of the progressive processing approach and in particular the conditions under which the CBDN centralized approach is more appropriate given certain PNs energy efficiency and software availability levels. Index Terms -Big data volume, big data variety, energy efficient networks, IP over WDM core networks, MILP, processing location optimization, software matching.

show abstract

On Modelling and Prediction of Total CPU Usage for Applications in MapReduce Environments

Cited by 19 publications

References 15 publications

Greening big data networks: velocity impact

Greening big data networks: velocity impact

Automated design of self-adaptive software with control-theoretical formal guarantees

Energy Efficient Big Data Networks: Impact of Volume and Variety

Contact Info

Product

Resources

About