Data-Intensive Workload Consolidation for the Hadoop Distributed File System

Moraveji, Reza; Taheri, Javid; Reza, M. Toufiq; Rizvandi, Nikzad Babaii; Zomaya, Albert Y.

doi:10.1109/grid.2012.25

Cited by 17 publications

(21 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since HCache utilizes a number of datanodes to enhance the parallel processing of the hot data, it is essential to validate the scalability of HCache .We test the Medium and Strong workload on HCache with 4,8,12,16,20,24 datanodes to show the system's scalability. We measure how long it takes to finish these two workloads.…”

Section: Scalabilitymentioning

confidence: 99%

“…However, it is worth noting that a number of research reports [5] [6][7] [8] from industry or academia indicate that many applications of MapReduce exhibit strong skewed access patterns. In real production environment, each application accesses a small number of files or blocks from the cluster, the overall access pattern within the Hadoop is non-uniform, some files are accessed more frequently than others.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

TS-Hadoop: Handling access skew in MapReduce by using tiered storage infrastructure

Wang

et al. 2014

2014 International Conference on Information and Communication Technology Convergence (ICTC)

View full text Add to dashboard Cite

Over the last few years, MapReduce systems has become popular for processing large-scale data sets and are increasingly being used in web indexing, data mining, and machine learning. Unlike simple application scenarios such as word count, many applications of MapReduce exhibit strong skewed access patterns in real production environment, the data access is non-uniform, often only a small portion of data are accessed far more frequently than others. Clearly, handling these hot data efficiently is quite critical to the overall performance of the MapReduce computation.In this paper, we present TS-Hadoop, a MapReduce system based on Apache Hadoop. The most significant feature of TSHadoop is that it utilizes tiered storage infrastructure, besides HDFS, TS-Hadoop also has a shared-disk cluster called HCache, it can be guaranteed that the data in HCache could be processed in highly parallel way. TS-Hadoop automatically distinguish hot and cold data based on current workload, and move them into HCache and HDFS respectively, the hot data in HCache could would be processed efficiently . Experiments show that the average execution time of MapReduce jobs in TS-Hadoop is much faster than traditional Hadoop platform when facing access skew workloads.

show abstract

Section: Scalabilitymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

TS-Hadoop: Handling access skew in MapReduce by using tiered storage infrastructure

Wang

et al. 2014

2014 International Conference on Information and Communication Technology Convergence (ICTC)

View full text Add to dashboard Cite

show abstract

“…A promising performance isolation must ensure a certain performance level, which is also predictable for service providers, among consolidated applications in such a way that operations of one VS do not affect the performance level of other services that concurrently running in the same physical machine 5,6 . Past studies demonstrated a lack of responsiveness in almost all VM management engines to adequately segregate the performance degradation, which is usually occurred due to the run‐time interference, among concurrently running VSs 7‐14 …”

Section: Introductionmentioning

confidence: 99%

Enhancing disk input output performance in consolidated virtualized cloud platforms using a randomized approximation scheme

HoseinyFarahabady

Taheri

Zomaya

et al. 2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

In a virtualized computer system with shared resources, consolidated virtual services (VSs) fiercely compete with each other to obtain the required capacity of resources, and this causes significant system's performance degradation. The performance of input output (I/O)‐bound applications running inside their own VS is mainly determined by the total time required to schedule every read/write request, plus the actual time needed by the device driver to complete the request. To achieve a right performance isolation of shared resources (e.g., the last level cache, memory bandwidth, and the disk buffer), it is essential to limit the performance degradation level among collocated applications, as simultaneously several I/O operations are requested by VSs, perhaps with different priorities. This article proposes a resource allocation controller that uses a fully polynomial‐time randomized approximation scheme to enable performance isolation of concurrent I/O requests in a shared system with multiple consolidated VSs. This controller uses a Monte Carlo sampling approach to measure and estimate the unknown attributes of operational requests originating from each VS. This is formalized as an optimization problem with the aim to minimize the degree of total quality of service (QoS) violation incidents in the entire platform. We associated a reward function to every working machine that represents the fulfillment degree of quality of service metric among all running VSs. The conducted comprehensive set of experiments showed that the proposed algorithm can reduce the QoS violation incidents by 32%, compared with the result which is obtained by employing the default resource allocation policy embedded in the existing Linux container layer.

show abstract

“…To optimize the performance of Hadoop cluster, we need to tune large number of configuration parameters [34]. Existing work has reported the impact of change in block size [18] and trying different replication strategies [26] to enhance throughput of the system. Each of these parameters can take more than one value.…”

Section: Hadoop Cluster Experimental Setup Design and Configuration Parameters Tuningmentioning

confidence: 99%

“…There have been a number of efforts to make current big data technologies more efficient, scalable and robust. These include improving load balancing and process scheduling capabilities across different environments [13] [14] [15] [16] [17] [18], improving security of data [19] [20] [21], opti-1 mized utilization of available data storage [22] [23], making of scalable and applied cloud technologies and developing performance matrices for distributed file systems [24][25][26][27][28]. Distributed computing paradigms, such as peers to peers, clusters, grids, and the cloud platforms, have been used for distributed data mining [29,30].…”

Section: Introductionmentioning

confidence: 99%

Resource and Performance Modelling of Hadoop Clusters Using Machine Learning

Tariq¹

View full text Add to dashboard Cite

<p>There is a huge and rapidly increasing amount of data being generated by social media, mobile applications and sensing devices. Big data is the term usually used to describe such data and is described in terms of the 3Vs - volume, variety and velocity. In order to process and mine such a massive amount of data, several approaches and platforms have been developed such as Hadoop. Hadoop is a popular open source distributed and parallel computing framework. It has a large number of configurable parameters which can be set before the execution of jobs to optimize the resource utilization and execution time of the clusters. These parameters have a significant impact on system resources and execution time. Optimizing the performance of a Hadoop cluster by tuning such a large number of parameters is a tedious task. Most current big data modeling approaches do not include the complex interaction between configuration parameters and the cluster environment changes such as use of different datasets or types of query. This makes it difficult to predict for example the execution time of a job or resource utilization of a cluster. Other attributes include configuration parameters, the structure of query, the dataset, number of nodes and the infrastructure used. Our first main objective was to design reliable experiments to understand the relationship between attributes. Before designing and implementing the actual experiment we applied Hazard and Operability (HAZOP) analysis to identify operational hazards. These hazards can affect normal working of cluster and execution of Hadoop jobs. This brainstorming activity improved the design and implementation of our experiments by improving the internal validity of the experiments. It also helped us to identify the considerations that must be taken into account for reliable results. After implementing our design, we characterized the relationship between different Hadoop configuration parameters, network and system performance measures. Our second main objective was to investigate the use of machine learning to model and predict the resource utilization and execution time of Hadoop jobs. Resource utilization and execution time of Hadoop jobs are affected by different attributes such as configuration parameters and structure of query. In order to estimate or predict either qualitatively or quantitatively the level of resource utilization and execution time, it is important to understand the impact of different combinations of these Hadoop job attributes. You could conduct experiments with many different combinations of parameters to uncover this but it is very difficult to run such a large number of jobs with different combinations of Hadoop job attributes and then interpret the data manually. It is very difficult to extract patterns from the data and give a model that can generalize for an unseen scenario. In order to automate the process of data extraction and modeling the complex behavior of different attributes of Hadoop job machine learning was used. Our decision tree based approach enabled us to systematically discover significant patterns in data. Our results showed that the decision tree models constructed for different resources and execution time were informative and robust. They were able to generalize over a wide range of minor and major environmental changes such as change in dataset, cluster size and infrastructure such as Amazon EC2. Moreover, the use of different correlation and regression techniques, such as M5P, Pearson's correlation and k-means clustering, confirmed our findings and provided further insight into the relationship of different attributes and with each other. M5P is a classification and regression technique that predicted the functional relationships among different job attributes. The use of k-means clustering allowed us to see the experimental runs that shows similar resource utilization and execution time. Statistical significance tests, were used to validate the significance of changes in results of different experimental runs, also showed the effectiveness of our resource and performance modelling and prediction method.</p>

show abstract

Data-Intensive Workload Consolidation for the Hadoop Distributed File System

Cited by 17 publications

References 15 publications

TS-Hadoop: Handling access skew in MapReduce by using tiered storage infrastructure

TS-Hadoop: Handling access skew in MapReduce by using tiered storage infrastructure

Enhancing disk input output performance in consolidated virtualized cloud platforms using a randomized approximation scheme

Resource and Performance Modelling of Hadoop Clusters Using Machine Learning

Contact Info

Product

Resources

About