Cross-MapReduce: Data transfer reduction in geo-distributed MapReduce

Marzuni, Seyed Saeed Mirpour; Savadi, Abdorreza; Toosi, Adel Nadjaran; Naghibzadeh, Mahmoud

doi:10.1016/j.future.2020.09.009

Cited by 6 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, some architectures such as Spark, Hadoop libraries, and MapReduce have been considered by different authors [103], [105], [108], though improvement is highly sorted for further studies.…”

Section: Open Research Directionsmentioning

confidence: 99%

“…(ii) Cross MapReduce Cross MapReduce is a technological framework by the Apache Hadoop ecosystem that provides data splitting into the distributed format, data mapping, shu ing, and classi cation to reduce document search [103]. This is essential in the processing of large climate change data because it is capable of processing all geo-distributed data.…”

Section: Data Types and Sourcesmentioning

confidence: 99%

“…This component is jointly utilized to minimize transferring of data volumes globally. For example, [103] proposed a Cross MapReduce framework to minimize transferred data volume and determine the number as well as the locations of global reducers. This, however, achieved a 40% reduction in the amount of data transfer over the Internet.…”

Section: Data Types and Sourcesmentioning

confidence: 99%

See 2 more Smart Citations

Recently Emerging Trends in Big Data Analytic Methods for Modeling and Combating Climate Change Effects

Ikegwu

Nweke

Alo

et al. 2023

Preprint

View full text Add to dashboard Cite

Big climate change data have become a pressing issue that organizations faced with methods to analyse data generated from various data types. However, storage, processing, and analysis of data generated from climate change activities are massive, which is challenging for the current algorithms to handle. Therefore, big data analytics methods are designed for significant data that is required to enhance seasonal change monitoring and understanding, ascertain the health risk of climate change, and improve the allocation, and utilisation of natural resources. This paper provides an outlook on big data analytic methods and describes how climate change and sustainability issues can be analysed through these methods. We extensively discuss big data analytic methods, strengths, and weaknesses. The purpose of analysing big climate change using these methods, the common datasets and implementation frameworks for climate change modeling using the big data analytics approach was also discussed. This big data analytics method is well timed to solve the inherent issues of data analysis and easy realization of sustainable development goals.

show abstract

Section: Open Research Directionsmentioning

confidence: 99%

Section: Data Types and Sourcesmentioning

confidence: 99%

Section: Data Types and Sourcesmentioning

confidence: 99%

See 1 more Smart Citation

Recently Emerging Trends in Big Data Analytic Methods for Modeling and Combating Climate Change Effects

Ikegwu

Nweke

Alo

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…However, this would lead to significant consumption of geo-distributed network resources to obtain those large-scale data from all involved sites; moreover, it demonstrates a high latency toward the completion of such jobs. Thus, many efforts have been made to improve the efficiency of these types of jobs by concurrently executing a data analytics job across multiple sites [2], [3], [4], [5].…”

Section: Introductionmentioning

confidence: 99%

Geo-Distributed IoT Data Analytics With Deadline Constraints Across Network Edge

Chen

Luo

Ren

et al. 2022

IEEE Internet Things J.

View full text Add to dashboard Cite

Owing to the advancement of the Internet of Things (IoT) and 5G mobile technologies, various IoT devices produce massive data which is usually transferred to nearby sites, such as edge nodes or datacenters. Many large-scale IoT applications need to analyze the data distributed across multiple sites to obtain final results. A dominant challenge of this type of data analytics is the heterogeneities of resource capacities across geodistributed sites. In this paper, we find that the resource capacity as well as the resource price differ among sites, and the price heterogeneity has a significant impact on geo-distributed IoT data analytics. Thus, each geo-distributed IoT data analytics job prefers to minimize the job execution cost while guaranteeing its deadline requirement under the resource constraints of involved sites. Specifically, we propose to jointly consider the resource heterogeneities of both capacity and price, and minimize the cost of each job before its deadline. We characterize this optimization problem as a Quadratically Constrained Quadratic Programming problem. To tackle such an NP-hard problem, we propose the MCGL method, which calculates a task placement solution by gradient adjustment strategy according to the remarkable negative correlation relationship between job completion time and job completion cost of geo-distributed IoT data analytics job. The task placement strategy can optimize resource cost with respect to the deadline requirement of any geo-distributed data analytics job. The trace-driven evaluations indicate that MCGL significantly reduces the total cost compared with existing methods; moreover, they satisfy the deadline constraints simultaneously.

show abstract

IDCOS: optimization strategy for parallel complex expression computation on big data

et al. 2021

View full text Add to dashboard Cite

Cross-MapReduce: Data transfer reduction in geo-distributed MapReduce

Cited by 6 publications

References 14 publications

Recently Emerging Trends in Big Data Analytic Methods for Modeling and Combating Climate Change Effects

Recently Emerging Trends in Big Data Analytic Methods for Modeling and Combating Climate Change Effects

Geo-Distributed IoT Data Analytics With Deadline Constraints Across Network Edge

IDCOS: optimization strategy for parallel complex expression computation on big data

Contact Info

Product

Resources

About