From the Cloud to the Atmosphere: Running MapReduce across Data Centers

Jayalath, Chamikara; Stephen, Julian James; Eugster, Patrick

doi:10.1109/tc.2013.121

Cited by 89 publications

(43 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Jayalath et al (2013) described the efficient way to process Bigdata across geographical distributed data centers. Li-Yung et al (2011) explained one optimization algorithm for cross Rack Optimization for Reducer program.…”

Section: Related Workmentioning

confidence: 99%

E-GENMR: Enhanced Generalized Query Processing using Double Hashing Technique through MapReduce in Cloud Database Management System

Malhotra¹,

Doja²,

Alam³

et al. 2017

Journal of Computer Science

View full text Add to dashboard Cite

Big Data, Cloud computing and Data Science is the booming future of IT industries. The common thing among all the new techniques is that they deal with not just Data but Big Data. Users store various kinds of data on cloud repositories. Cloud Database Management System deals with these large sets of data. Cloud Database service provider deals with many obstacles while providing various service. Amongst all the challenges processing of large amount of data, interoperability and security are the major concerns that are explained in this study. Enhanced Generalized Query Processing through MapReduce (E-GENMR) is a prototype model that provides solution for these problems. Firstly, traditional approaches are not suitable for processing such gigantic amount of data as they are not able to handle such amount of data. Various solutions have been developed such as Hadoop, MapReduce Programming codes, HIVE, PIG etc. but these technologies don't provide solution for these problems at the same time and moreover users are not compatible with these latest technologies like MapReduce codes. E-GENMR provides interoperability as it takes queries written in various RDBMS forms like SQL Server, ORACLE, DB2, MYSQL and convert into MapReduce codes as they are considered to be the efficient way for processing large data. Secondly, Client's data is stored in encrypted form and processing is done on this data hence it ensures the security aspect. Indexing plays a very important role in processing queries, in E-GENMR indexing is implemented using closed double hashing technique. We compared various query processing time of E-GENMR for encrypted data and unencrypted data. A comparison of various queries has been done to evaluate the performance of E-GENMR with latest techniques like Hadoopdb, SQLMR, HIVE and PIG and it has been concluded that E-GENMR shows better performance.

show abstract

Section: Related Workmentioning

confidence: 99%

E-GENMR: Enhanced Generalized Query Processing using Double Hashing Technique through MapReduce in Cloud Database Management System

Malhotra¹,

Doja²,

Alam³

et al. 2017

Journal of Computer Science

View full text Add to dashboard Cite

show abstract

“…The authors in [14] aimed to keep the communication cost to a minimum by satisfying as many big data queries as possible over a number of time slots. The authors in [15] developed a framework to perform a sequence of MapReduce jobs in Geo-distributed DCs where the processing of jobs is optimized according to time and monetary cost. In [16], the authors developed an energy efficient cloud computing framework in IP over WDM core networks.…”

Section: Introductionmentioning

confidence: 99%

Energy Efficient Big Data Networks: Impact of Volume and Variety

Al-Salim

Lawey

El-Gorashi

et al. 2018

IEEE Trans. Netw. Serv. Manage.

View full text Add to dashboard Cite

Abstract-In this article, we study the impact of big data's volume and variety dimensions on Energy Efficient Big Data Networks (EEBDN) by developing a Mixed Integer Linear Programming (MILP) model to encapsulate the distinctive features of these two dimensions. Firstly, a progressive energy efficient edge, intermediate, and central processing technique is proposed to process big data's raw traffic by building processing nodes (PNs) in the network along the way from the sources to datacenters. Secondly, we validate the MILP operation by developing a heuristic that mimics, in real time, the behaviour of the MILP for the volume dimension. Thirdly, we test the energy efficiency limits of our green approach under several conditions where PNs are less energy efficient in terms of processing and communication compared to data centers. Fourthly, we test the performance limits in our energy efficient approach by studying a "software matching" problem where different software packages are required to process big data. The results are then compared to the Classical Big Data Networks (CBDN) approach where big data is only processed inside centralized data centers. Our results revealed that up to 52% and 47% power saving can be achieved by the EEBDN approach compared to the CBDN approach, under the impact of volume and variety scenarios, respectively. Moreover, our results identify the limits of the progressive processing approach and in particular the conditions under which the CBDN centralized approach is more appropriate given certain PNs energy efficiency and software availability levels. Index Terms -Big data volume, big data variety, energy efficient networks, IP over WDM core networks, MILP, processing location optimization, software matching.

show abstract

“…The authors in [4] proposed a MapReduce framework to locally process as much data as possible on multiple IoT nodes rather than transmitting the raw data to datacentres (DCs). The authors in [5] presented a processing system for executing a sequence of MapReduce jobs on Geo-distributed datacentres where the treating of jobs is optimized according to time and pecuniary cost. The authors in [6] proposed a dynamic bulk data transfer framework in geo-distributed data canters and planned its design and algorithms depending on Software Defined Network (SDN) architecture.…”

Section: Introductionmentioning

confidence: 99%

Greening big data networks: Volume impact

Al-Salim

Ali

Lawey

et al. 2016

2016 18th International Conference on Transparent Optical Networks (ICTON)

View full text Add to dashboard Cite

Tremendous volumes generated by big data applications are starting to overwhelm data centers and networks. Traditional research efforts have determined how to process these vast volumes of data inside datacenters. Nevertheless, slight attention has addressed the increase in power consumption resulting from transferring these gigantic volumes of data from the source to destination (datacenters). An efficient approach to address this challenge is to progressively processing large volumes of data as close to the source as possible and transport the reduced volume of extracted knowledge to the destination. In this article, we examine the impact of processing different big data volumes on network power consumption in a progressive manner from source to datacenters. Accordingly, a noteworthy decrease for data transferred is achieved which results in a generous reduction in network power consumption. We consider different volumes of big data chunks. We introduce a Mixed Integer Linear Programming model (MILP) to optimize the processing locations of these volumes of data and the locations of two datacenters. The results show that serving different big data volumes of uniform distribution yields higher power saving compared to the volumes of chunks with fixed size. Therefore, we obtain an average network power saving of 57%, 48%, and 35% when considering the volumes of 10-220 (uniform) Gb, 110 Gb, and 50 Gb per chunk, respectively, compared to the conventional approach where all these chunks are processed inside datacenters only.

show abstract

From the Cloud to the Atmosphere: Running MapReduce across Data Centers

Cited by 89 publications

References 7 publications

E-GENMR: Enhanced Generalized Query Processing using Double Hashing Technique through MapReduce in Cloud Database Management System

E-GENMR: Enhanced Generalized Query Processing using Double Hashing Technique through MapReduce in Cloud Database Management System

Energy Efficient Big Data Networks: Impact of Volume and Variety

Greening big data networks: Volume impact

Contact Info

Product

Resources

About