HSim: A MapReduce simulator in enabling Cloud Computing

Liu, Yang; Li, Maozhen; Alham, Nasullah Khalid; Hammoud, Suhel

doi:10.1016/j.future.2011.05.007

Cited by 56 publications

(28 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We did not use existing Hadoop simulators [10], [11] because the majority of them are developed to study job scheduling or resource management stuffs, and are unable to simulate multicast network traffic. The ns-2 based MRPerf [12] is the only candidate that can simulate the network traffic of Hadoop system in detail.…”

Section: Simulation Studymentioning

confidence: 99%

Multicast-based replication for Hadoop HDFS

Hong

2015

2015 IEEE/ACIS 16th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distribu

View full text Add to dashboard Cite

The Hadoop HDFS is a popular open-source distributed storage system, which serves as the foundation of many important big-data technologies. The performance of data replication is crucial to HDFS, since it accounts for a major portion of network traffic in the entire cluster. In this research, we propose to enable multicast-based replication, which is expected to use less network bandwidth than the native TCP-based pipelined replication method. We developed a congestion-controlled reliable multicast socket (the CCRMSocket) for HDFS and evaluated its performance with our multi-rack test platform. The experimental result shows that our multicast implementation can effectively save bandwidth and peacefully coexist with TCP traffic. We also developed a simulator (the HFlowSim) to further study the impact of multicast-based replication to a large-scale Hadoop system. The simulation result suggests that multicast-based replication can systematically improve a Hadoop system by accelerating the big jobs.978-1-4799-8676-7/15/$31.

show abstract

Section: Simulation Studymentioning

confidence: 99%

Multicast-based replication for Hadoop HDFS

Hong

2015

2015 IEEE/ACIS 16th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distribu

View full text Add to dashboard Cite

show abstract

“…While Google kept its MapReduce system proprietary, Apache Hadoop [1] is the most popular open-source implementation of MapReduce framework. According to a report from Gartner [3], 65% of the packaged data analytic applications will be built on Hadoop [33,26,18,9,5], YARNsim is advantageous in that it is a parallel simulation system that is potentially capable of simulating extremescale Hadoop YARN systems. Similar systems are constrained, for example, MRperf uses ns-2 as its network module.…”

Section: Hadoopmentioning

confidence: 99%

FatTreeSim

Liu

Haider

Sun

et al. 2015

Proceedings of the 3rd ACM SIGSIM Conference on Principles of Advanced Discrete Simulation

View full text Add to dashboard Cite

Fat-tree topologies have been widely adopted as the communication network in data centers in the past decade. Nowadays, high-performance computing (HPC) system designers are considering using fat-tree as the interconnection network for the next generation supercomputers. For extreme-scale computing systems like the data centers and supercomputers, the performance is highly dependent on the interconnection networks. In this paper, we present FatTreeSim, a PDES-based toolkit consisting of a highly scalable fat-tree network model, with the goal of better understanding the design constraints of fat-tree networking architectures in data centers and HPC systems, as well as evaluating the applications running on top of the network. FatTreeSim is designed to model and simulate large-scale fat-tree networks up to millions of nodes with protocol-level fidelity. We have conducted extensive experiments to validate and demonstrate the accuracy, scalability and usability of FatTreeSim. On Argonne Leadership Computing Facility's Blue Gene/Q system, Mira, FatTreeSim is capable of achieving a peak event rate of 305 M/s for a 524,288-node fat-tree model with a total of 567 billion committed events. The strong scaling experiments use up to 32,768 cores and show a near linear scalability. Comparing with a small-scale physical system in Emulab, FatTreeSim can accurately model the latency in the same fat-tree network with less than 10% error rate for most cases. Finally, we demonstrate FatTreeSim's usability through a case study in which FatTreeSim serves as the network module of the YARNsim system, and the error rates for all test cases are less than 13.7%.

show abstract

“…Depending on the service provided, the cloud can offer three different levels of abstraction: software as a service, platform as a service and infrastructure as a service. With software as a service the user does not need any additional specialized software or hardware to run the application, it just runs on servers in the cloud and the user cant start using with sometimes as little as a web browser or mobile app and an internet connection [7], [11].…”

Section: Background and Related Work A Cloud Computing And Simulmentioning

confidence: 99%

Cloud simulation under fault constraints

Bosilca

Nita

Pop

et al. 2014

2014 IEEE 10th International Conference on Intelligent Computer Communication and Processing (ICCP)

View full text Add to dashboard Cite

As Cloud Computing offers support to more and more complex applications, the need to verify and validate computing models under fault constraints becomes more important, aiming to ensure applications performance. Doing this experimental validation in the early development phase and with small costs require a cloud simulation tool. An extensible framework for Cloud simulation and modeling is CloudSim. This paper proposes a new module for CloudSim consisting of a fault injector based on a specification language. The aim is to assist simulation to be more realistic and includes concrete conditions and constraints. The impact is on testing Cloud applications and help test fault tolerant applications by specifying defect patterns and failing components. The evaluation of the fault injection module is done by measuring the behavior and performance of a tool based on CloudSim, named CloudAnalyst. Several metrics are determined and measured for experimental validation, and conclusions are drawn.

show abstract

HSim: A MapReduce simulator in enabling Cloud Computing

Cited by 56 publications

References 6 publications

Multicast-based replication for Hadoop HDFS

Multicast-based replication for Hadoop HDFS

FatTreeSim

Cloud simulation under fault constraints

Contact Info

Product

Resources

About