Scalable Multi-purpose Network Representation for Large Scale Distributed System Simulation

Abstract

Abstract-Conducting experiments in large-scale distributed systems is usually time-consuming and labor-intensive. Uncontrolled external load variation prevents to reproduce experiments and such systems are often not available to the purpose of research experiments, e.g., production or yet to deploy systems. Hence, many researchers in the area of distributed computing rely on simulation to perform their studies. However, the simulation of large-scale computing systems raises several scalability issues, in terms of speed and memory. Indeed, such systems now comprise millions of hosts interconnected through a complex network and run billions of processes. Most simulators thus trade accuracy for speed and rely on very simple and easy to implement models. However, the assumptions underlying these models are often questionable, especially when it comes to network modeling.In this paper, we show that, despite a widespread belief in the community, achieving high scalability does not necessarily require to resort to overly simple models and ignore important phenomena. We show that relying on a modular and hierarchical platform representation, while taking advantage of regularity when possible, allows us to model systems such as data and computing centers, peer-to-peer networks, grids, or clouds in a scalable way. This approach has been integrated into the opensource SIMGRID simulation toolkit. We show that our solution allows us to model such systems much more accurately than other state-of-the-art simulators without trading for simulation speed. SIMGRID is even sometimes orders of magnitude faster.

Simulating Application Workflows and Services Deployed on the European Grid Infrastructure

“…SimGrid [6], [33] is a toolkit that provides core functionalities for the simulation of distributed applications in heterogeneous distributed environments. This section describes the functionalities used in our simulation.…”

“…The routing between the elements of an AS has to be explicitly defined. Network is simulated using a fluid network model [33]. Hosts define physical resources with computing capabilities, one or more mailboxes to enable communication, and some private data that can be only accessed by local processes.…”

Toward Better Simulation of MPI Applications on Ethernet/TCP Networks

“…Since SimGrid, on which SMPI is based, is a versatile simulator, incorporating further levels of hierarchy or more complex interconnections if needed would be easy [7]. Figure 5.…”

Simizer

“…The motivation behind this work is the need of a simple and reliable testing framework for large (> 100 machines) distributed systems load balancing and consistency / data synchronization protocols. Existing simulation tools and libraries such as OptorSim, CloudSim and SimGrid [2][3][4] do not provide simple means for simulating data consistency issues in large scale systems. The approach developed in the Simizer simulator is to provide a simple programming interface for modeling distributed applications and data storage systems.…”