Towards optimization of Hadoop Map reduce jobs on cloud

Lakshmi, Adepu Sree; BalRaju, M.; Chandra, N. Subhash

doi:10.1109/cast.2016.7914976

Cited by 3 publications

(4 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dealing with a large volume of data, several studies revealed the use of Hadoop for processing big data. It is vital that such large number of datasets should be adjusted to handle the dynamic process of workload through the existing computer resources [12,14,15]. A Hadoop cluster with a fixed size is shown to have problems when processing the streaming data and it is also not cost-effective for the cloud resources since it does not use fully its resources when the data load on the cluster is low.…”

Section: Related Workmentioning

confidence: 99%

“…However, this approach is not as flexible as AWS and is also not cost-effective [9]. Additionally, in the traditional Hadoop cluster with a First-In First-Out (FIFO) scheduler, all task slots for a job in the cluster blocks other jobs to use the resources until the current scheduled job finishes, which may cause a bottleneck in processing the data [14], especially when dealing with massive data (e.g., Twitter) arriving at the same time. As one of the biggest social platforms, Twitter provides a huge data set which can be streamed for sentiment analysis.…”

Section: Related Workmentioning

confidence: 99%

“…This work aims to provide an automatic scaling system which predicts the sentiment of the tweets of the people posted on social media. Hadoop clusters are employed to process the data and MapReduce programming is used to analyze the sentiment statistics on the Twitter dataset due to its capability in performing distributed and parallel processing on large data sets [10,14]. The data from different universities in the United Kingdom is collected by following a Twitter API link to perform sentiment analysis.…”

Section: Test Case Of the Proposed Ceas Frameworkmentioning

confidence: 99%

“…However, selecting a cloud infrastructure for auto-scaling and optimising the virtual resources for such rapid data growth is a challenging task. Determining scaling policies on new virtual machines as well as setting up a threshold that can automatically configure the computing resources on a Hadoop cluster have therefore attracted more research interests [12][13][14][15].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

An Auto-Scaling Framework for Analyzing Big Data in the Cloud Environment

et al. 2019

View full text Add to dashboard Cite

Processing big data on traditional computing infrastructure is a challenge as the volume of data is large and thus high computational complexity. Recently, Apache Hadoop has emerged as a distributed computing infrastructure to deal with big data. Adopting Hadoop to dynamically adjust its computing resources based on real-time workload is itself a demanding task, thus conventionally a pre-configuration with adequate resources to compute the peak data load is set up. However, this may cause a considerable wastage of computing resources when the usage levels are much lower than the preset load. In consideration of this, this paper investigates an auto-scaling framework on cloud environment aiming to minimise the cost of resource use by automatically adjusting the virtual nodes depending on the real-time data load. A cost-effective auto-scaling (CEAS) framework is first proposed for an Amazon Web Services (AWS) Cloud environment. The proposed CEAS framework allows us to scale the computing resources of Hadoop cluster so as to either reduce the computing resource use when the workload is low or scale-up the computing resources to speed up the data processing and analysis within an adequate time. To validate the effectiveness of the proposed framework, a case study with real-time sentiment analysis on the universities’ tweets is provided to analyse the reviews/tweets of the people posted on social media. Such a dynamic scaling method offers a reference to improving the Twitter data analysis in a more cost-effective and flexible way.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Test Case Of the Proposed Ceas Frameworkmentioning

confidence: 99%