A comprehensive view of Hadoop research—A systematic literature review

Polato, Ivanilton; Ré, Reginaldo; Goldman, Alfredo; Kon, Fábio

doi:10.1016/j.jnca.2014.07.022

Cited by 116 publications

(61 citation statements)

References 120 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The targeted performance metric in these studies is the job response time, which is in fact an integral part of the business model of MapReduce based query systems such as [32] and time priced computing clouds such as Amazon's EC2 [1]. For an overview on works that optimize the performance of MapReduce systems see the survey article [33]. Using a similar idea as in [5], the authors of [37] derive asymptotic results on the response time distribution in the case of renewal arrivals; such results are further used to understand the impact of different scheduling models in the reduce phase of MapReduce.…”

Section: Related Workmentioning

confidence: 99%

Stochastic bounds in Fork–Join queueing systems under full and partial mapping

2016

View full text Add to dashboard Cite

In a Fork-Join (FJ) queueing system an upstream fork station splits incoming jobs into N tasks to be further processed by N parallel servers, each with its own queue; the response time of one job is determined, at a downstream join station, by the maximum of the corresponding tasks' response times. This queueing system is useful to the modelling of multi-service systems subject to synchronization constraints, such as MapReduce clusters or multipath routing. Despite their apparent simplicity, FJ systems are hard to analyze.This paper provides the first computable stochastic bounds on the waiting and response time distributions in FJ systems under full (bijective) and partial (injective) mapping of tasks to servers. We consider four practical scenarios by combining 1a) renewal and 1b) non-renewal arrivals, and 2a) non-blocking and 2b) blocking servers. In the case of non-blocking servers we prove that delays scale as O(log N ), a law which is known for first moments under renewal input only. In the case of blocking servers, we prove that the same factor of log N dictates the stability region of the system. Simulation results indicate that our bounds are tight, especially at high utilizations, in all four scenarios. A remarkable insight gained from our results is that, at moderate to high utilizations, multipath routing "makes sense" from a queueing perspective for two paths only, i.e., response times drop the most when N = 2; the technical explanation is that the resequencing (delay) price starts to quickly dominate the tempting gain due to multipath transmissions.

show abstract

Section: Related Workmentioning

confidence: 99%

Stochastic bounds in Fork–Join queueing systems under full and partial mapping

2016

View full text Add to dashboard Cite

show abstract

“…The targeted performance metric in these studies is the job response time, which is in fact an integral part of the business model of MapReduce based query systems such as [27] and time priced computing clouds such as Amazon's EC2 [1]. For an overview on works that optimize the performance of MapReduce systems see the survey article [28]. Using a similar idea as in [4], the authors of [32] derive asymptotic results on the response time distribution in the case of renewal arrivals; such results are further used to understand the impact of different scheduling models in the reduce phase of MapReduce.…”

Section: Related Workmentioning

confidence: 99%

Computable Bounds in Fork-Join Queueing Systems

Rizk

Poloczek

Ciucu

2015

Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems

View full text Add to dashboard Cite

In a Fork-Join (FJ) queueing system an upstream fork station splits incoming jobs into N tasks to be further processed by N parallel servers, each with its own queue; the response time of one job is determined, at a downstream join station, by the maximum of the corresponding tasks' response times. This queueing system is useful to the modelling of multiservice systems subject to synchronization constraints, such as MapReduce clusters or multipath routing. Despite their apparent simplicity, FJ systems are hard to analyze.This paper provides the first computable stochastic bounds on the waiting and response time distributions in FJ systems. We consider four practical scenarios by combining 1a) renewal and 1b) non-renewal arrivals, and 2a) non-blocking and 2b) blocking servers. In the case of non-blocking servers we prove that delays scale as O(log N ), a law which is known for first moments under renewal input only. In the case of blocking servers, we prove that the same factor of log N dictates the stability region of the system. Simulation results indicate that our bounds are tight, especially at high utilizations, in all four scenarios. A remarkable insight gained from our results is that, at moderate to high utilizations, multipath routing "makes sense" from a queueing perspective for two paths only, i.e., response times drop the most when N = 2; the technical explanation is that the resequencing (delay) price starts to quickly dominate the tempting gain due to multipath transmissions.

show abstract

“…Hadoop has changed constantly and considerably in 59 releases over six years of development. The Hadoop framework has been widely adopted by both the industry and research communities [9]. Due to Hadoop's key concept of parallel and distributed abstraction, it is recently adopted by several big companies such as Facebook, Ebay, Yahoo, Amazon, Adobe and more.…”

Section: Bug-source Software Selectionmentioning

confidence: 99%

Concurrency bugs in open source software: a case study

Asadollah

Sundmark

Eldh

et al. 2017

J Internet Serv Appl

View full text Add to dashboard Cite

Concurrent programming puts demands on software debugging and testing, as concurrent software may exhibit problems not present in sequential software, e.g., deadlocks and race conditions. In aiming to increase efficiency and effectiveness of debugging and bug-fixing for concurrent software, a deep understanding of concurrency bugs, their frequency and fixing-times would be helpful. Similarly, to design effective tools and techniques for testing and debugging concurrent software, understanding the differences between non-concurrency and concurrency bugs in real-word software would be useful. This paper presents an empirical study focusing on understanding the differences and similarities between concurrency bugs and other bugs, as well as the differences among various concurrency bug types in terms of their severity and their fixing time, and reproducibility. Our basis is a comprehensive analysis of bug reports covering several generations of five open source software projects. The analysis involves a total of 11860 bug reports from the last decade, including 351 reports related to concurrency bugs. We found that concurrency bugs are different from other bugs in terms of their fixing time and severity while they are similar in terms of reproducibility. Our findings shed light on concurrency bugs and could thereby influence future design and development of concurrent software, their debugging and testing, as well as related tools.

show abstract

A comprehensive view of Hadoop research—A systematic literature review

Cited by 116 publications

References 120 publications

Stochastic bounds in Fork–Join queueing systems under full and partial mapping

Stochastic bounds in Fork–Join queueing systems under full and partial mapping

Computable Bounds in Fork-Join Queueing Systems

Concurrency bugs in open source software: a case study

Contact Info

Product

Resources

About