Optimization of Hadoop MapReduce Model in cloud Computing Environment

Sontakke, Vaishali; Dayanand, R B

doi:10.1109/icssit46314.2019.8987823

Cited by 9 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apache Spark is a fast and universal computational engine designed for large-scale data processing, the same as Hadoop, whose computational model is based on MapReduce [1,2] . In 2012, UC Berkeley's AMPLab developed and open-sourced a new big data processing framework, Spark, whose core ideas include the following two aspects: on the one hand, the input and output of the big data processing framework and the intermediate data are abstractly modeled and represented in a unified data structure, which is named Resilient Distributed Dataset (Resilient Distributed Dataset (RDD).…”

Section: Sparkmentioning

confidence: 99%

Research on Rule Matching Model Based on Spark

Xiong

Jiang

2022

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

Rule engines are widely used in engineering and academic fields because they can flexibly separate facts and rules from the rule matching process. Since the rule matching process is very time-consuming, and the traditional rule matching uses single-computer operation when many facts and rules exceed the computer’s memory and computational capacity limit, it will cause the application to crash and paralyze. To solve the above problems, this paper investigates the Spark framework and Rete algorithm to take advantage of Spark’s in-memory computation to alleviate the time-consuming problem of the traditional rule matching process. A high-performance distributed rule matching model is designed by combining the actual rule matching scenario and the development process. In addition, according to the form of rules and facts in the actual scenario, this paper effectively divides the matching process and improves the scalability of the rule matching model.

show abstract

Section: Sparkmentioning

confidence: 99%

Research on Rule Matching Model Based on Spark

Xiong

Jiang

2022

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

show abstract

“…MapReduce is the main programming model of cloud computing, which is used to process and generate large datasets for various tasks in the real world [2]. Dayanand and others put forward an optimized HPMR (Hadoop MapReduce) model, which balances the performance between I/O system and CPU [3]. Liu Jun and others proposed a configuration parameter adjustment method based on feature selection algorithm, which improved the working efficiency of MapReduce in Hadoop [4].…”

Section: Introductionmentioning

confidence: 99%

Research on Intelligentization of Cloud Computing Programs Based on Self-awareness

Liu

Gao

Luo

2023

International Journal of Advanced Network, Monitoring and Controls

View full text Add to dashboard Cite

Through the research of MapReduce programming framework of cloud computing, the current MapReduce program only solves specific problems, and there is no design experience or design feature summary of MapReduce program, let alone formal description and experience inheritance and application of knowledge base. In order to solve the problem of intelligent cloud computing program, a general MapReduce program generation method is designed. This paper proposes the architecture of intelligent cloud computing by studying AORBCO model and combining cloud computing technology. According to the behavior control mechanism in AORBCO model, a program generation method of MapReduce in intelligent cloud computing is proposed. This method will extract entity information in input data set and entity information in knowledge base in intelligent cloud computing for similarity calculation, and extract the entity in the top order as key key-value pair information in intelligent cloud computing judgment data set. The data processing types are divided, and then aligned with each specific MapReduce capability, and the MapReduce program generation experiment is verified in the AORBCO model development platform. The experiment shows that the complexity of big data MapReduce program code is simplified, and the generated code execution efficiency is good.

show abstract

“…Varying users can execute various querying the data and so has the ability to pull out valuable results from the filtered data and afterwards rate the results based on the dimensions they need. These studies allow users to discover the current business trends with their plans they can make changes  Complexity: The complexity of a system is measured by the extent of interdependence within enormous data structures, and a tiny change in one or a few pieces can lead to extremely huge changes or a little change that causes a shift in only a portion of the system broad reaching or disseminating impacts on the system, or no alterations at all (Katal, Wazid, & Goudar, 2013) [3] (possibly very large).…”

Section: Introductionmentioning

confidence: 99%

An Efficient Approach for Bigdata Security Based on Hadoop System Using Cryptographic Techniques

Gattoju¹,

Nagalakshmi²

2021

INDJCSE

View full text Add to dashboard Cite

When relational database systems could no longer keep up with the huge amounts of unstructured data created by organizations, social media, and all other data-generating sources, big data came into being. The amount of data being added every day, together with Hadoop, makes for an urgent and growing need for more data processing solutions. The MapReduce programming model is one common approach for processing and handling huge amounts of data, especially when used to big data research. As HDFS, a distributed, scalable, and portable file system constructed in Java for the Hadoop architecture, is already useful, it is noteworthy that it is built using Java technology. This computing environment suffers from two issues. First, when intruders access the system, they can steal or corrupt the data stored in the system. The AES encryption mechanism has been implemented in HDFS to safeguard the security of data stored in HDFS. Some data saved in HDFS can be secured with the application of AES encryption technique. I conducted an extensive research on security challenges around large data in the context of Hadoop, along with numerous solutions and technologies utilized to secure it.

show abstract

Optimization of Hadoop MapReduce Model in cloud Computing Environment

Cited by 9 publications

References 21 publications

Research on Rule Matching Model Based on Spark

Research on Rule Matching Model Based on Spark

Research on Intelligentization of Cloud Computing Programs Based on Self-awareness

An Efficient Approach for Bigdata Security Based on Hadoop System Using Cryptographic Techniques

Contact Info

Product

Resources

About