Variable-Sized Map and Locality-Aware Reduce on Public-Resource Grids

Chen, Po‐Cheng; Su, Yen-Liang; Chang, Jyh-Biau; Shieh, Ce-Kuen

doi:10.1007/978-3-642-13067-0_27

Cited by 8 publications

(6 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In large-scale dataintensive applications like distributed data processing almost provides minimal abstractions which is also used to hide architectural details. Thus automatically parallelizes computation [17], and supports transparent fault tolerance via data repetition, replication and the copy of data continuously used in computation [3]. A important characteristic need of MapReduce [4,6,12] is effortlessness that permit programmers to inscribe functional-style and to code the execution of job in moderate and reliable method.…”

Section: Priority Based Classical Data Encapsulated Scheduling For Nementioning

confidence: 99%

“…Such kind of issues is data reduction, which requires to be implemented with very large-scale cluster data de-duplication systems [13]. The similarity-based deduplication scheme uses [3,4] to optimizes the exclusion procedure by bearing in mind the locality and resemblance of data points that are occurred in together the inter and intra node circumstances. Today's technology like Hadoop and Map Reduce [1,2,8] are facing such kind of redundancy elimination while data utilization.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Priority Based Classical Data Encapsulated Scheduling For New Era in Computing Environment

Gavaskar¹,

Rajkumar²,

Sudheer³

2019

IJITEE

View full text Add to dashboard Cite

In computational environment the data processing and data transactions are major issue, to overcome this problem of processing and reduce the band width in distributed models. The abstract mentioned here clearly state the novelty of the work regarding to data processing and data utilization. This paper shows that the data provided between nodes that are involved in clusters and that is useful to the data utilization schematic in all kinds of clusters. The proposed approach arranges data into data tables and distributes them transversely nodes in a cluster. It recommends jobs to be processed by the cluster then transfers packaged and encapsulated data into nodes to process the data in parallel. This technique employs an incremental data scheduled computation way that keep away from the costly enumeration of data pattern matching necessary by cluster methods. It improves the data locality by forwarding data to the job supporting cluster. This abstraction is enthused by the data processing and data reduce primitives presented in the various job processing environment and many other functional applications used in data grid, data intensive approaches. The Architectural interface is provided in between clusters and is used to achieve high performance on large clusters of commodity PCs. This approach reduces the data request and its processing when it is signed into data clusters. The proposed approach arranges data into data tables and distributes them across nodes in a cluster. It recommend jobs to be processed by the cluster then transfers packaged and encapsulated data into nodes to progression the data in parallel. It improves the data locality by forwarding data to the job supporting cluster. The proposed approach in this paper is helpful where number of nodes involve in cluster is always increasing due to the high end computations that are involved in distributed environment. ), otherwise Constraint satisfactions Algorithm:Step -1: initialize variable of job table J ET = 0 J p =0; J ET  J p ( J ET defines J p ) Step-2: Tolerate data set with D t ( d i ,d p , d tv )

show abstract

Section: Priority Based Classical Data Encapsulated Scheduling For Nementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Priority Based Classical Data Encapsulated Scheduling For New Era in Computing Environment

Gavaskar¹,

Rajkumar²,

Sudheer³

2019

IJITEE

View full text Add to dashboard Cite

show abstract

“…Ussop [2] employs MapReduce on public-resource grids and suggests variable-size map tasks. LARTS [9] attempts to collocate reduce tasks with the maximum required data.…”

Section: Related Workmentioning

confidence: 99%

“…In our experiments, we observed that the total intermediate output (or total reduce input) size is sometimes equal to the total input size of all map tasks (e.g. sort) or even larger (e.g., 44.2% for K-means) 2 . Similar observations were reached in [10], [13].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Center-of-Gravity Reduce Task Scheduling to Lower MapReduce Network Traffic

Hammoud

Rehman

Sakr

2012

2012 IEEE Fifth International Conference on Cloud Computing

View full text Add to dashboard Cite

MapReduce is by far one of the most successful realizations of large-scale data-intensive cloud computing platforms. MapReduce automatically parallelizes computation by running multiple map and/or reduce tasks over distributed data across multiple machines.Hadoop is an open source implementation of MapReduce. When Hadoop schedules reduce tasks, it neither exploits data locality nor addresses partitioning skew present in some MapReduce applications. This might lead to increased cluster network traffic. In this paper we investigate the problems of data locality and partitioning skew in Hadoop. We propose Center-of-Gravity Reduce Scheduler (CoGRS), a locality-aware skew-aware reduce task scheduler for saving MapReduce network traffic. In an attempt to exploit data locality, CoGRS schedules each reduce task at its center-ofgravity node, which is computed after considering partitioning skew as well. We implemented CoGRS in Hadoop-0.20.2 and tested it on a private cloud as well as on Amazon EC2. As compared to native Hadoop, our results show that CoGRS minimizes off-rack network traffic by averages of 9.6% and 38.6% on our private cloud and on an Amazon EC2 cluster, respectively. This reflects on job execution times and provides an improvement of up to 23.8%.

show abstract