Adding Storage Simulation Capacities to the SimGrid Toolkit: Concepts, Models, and API

Abstract

Abstract-For each kind of distributed computing infrastructures, i.e., clusters, grids, clouds, data centers, or supercomputers, storage is a essential component to cope with the tremendous increase in scientific data production and the ever-growing need for data analysis and preservation. Understanding the performance of a storage subsystem or dimensioning it properly is an important concern for which simulation can help by allowing for fast, fully repeatable, and configurable experiments for arbitrary hypothetical scenarios. However, most simulation frameworks tailored for the study of distributed systems offer no or little abstractions or models of storage resources.In this paper, we detail the extension of SimGrid, a versatile toolkit for the simulation of large-scale distributed computing systems, with storage simulation capacities. We first define the required abstractions and propose a new API to handle storage components and their contents in SimGrid-based simulators. Then we characterize the performance of the fundamental storage component that are disks and derive models of these resources. Finally we list several concrete use cases of storage simulations in clusters, grids, clouds, and data centers for which the proposed extension would be beneficial.

WRENCH: A Framework for Simulating Workflow Management Systems

“…SimGrid is open source and freely available, has been stable for many years, is actively developed, has a sizable user community, and has provided simulation results for over 350 research publications since its inception. SimGrid has also been the object of many invalidation and validation studies [42]- [46], and its simulation models have been shown to provide compelling advantages over other simulation frameworks in terms of both accuracy and scalability [33]. Finally, most SimGrid simulations can be executed in minutes on a standard laptop computer, making it possible to perform large numbers of simulations quickly with minimal compute resource expenses.…”

“…We claim that advances in simulation capabilities in the field have made it possible to simulate WMSs that execute large workflows on large-scale platforms accessible via diverse CI services in a way that is accurate (via validated simulation models), scalable (fast execution and low memory footprint), and expressive (ability to describe arbitrary platforms, complex WMSs, and complex software infrastructure). In this work, we build on the existing open-source SimGrid simulation framework [33], [41], which has been one of the drivers of the above advances and whose simulation models have been extensively validated [42]- [46], to develop a WMS simulation framework called WRENCH [47]. More specifically, this work makes the following contributions: 1) We justify the need for WRENCH and explain how it improves on the state of the art.…”

A Data Model for Supplying a Data Center with Several Energy Sources

“…Simgrid [3] is a simulator for distributed applications heterogeneous in distributed environments. It allows the execution of algorithms for scheduling a Data Center [4], the simulation of storage in a Data Center [5] and the use of the DVFS (Dynamic Voltage and Frequency Scaling) in a Data Center [6]. This simulator is in the improvement of the actual application in the field of grids.…”

Developing accurate and scalable simulators of production workflow management systems with WRENCH

“…We claim that advances in simulation capabilities in the field have made it possible to simulate WMSs that execute large workflows using diverse CI services deployed on large-scale platforms in a way that is accurate (via validated simulation models), scalable (fast execution and low memory footprint), and expressive (ability to describe arbitrary platforms, complex WMSs, and complex software infrastructure). In this work, we build on the existing open-source SimGrid simulation framework [33,41], which has been one of the drivers of the above advances and whose simulation models have been extensively validated [42,43,44,45,46], to develop a WMS simulation framework called WRENCH [47]. More specifically, this work makes the following contributions 1 : 1 A preliminary shorter version of this paper appears in the proceedings 1.…”

“…SimGrid is open source and freely available, has been stable for many years, is actively developed, has a sizable user community, and has provided simulation results for over 350 research publications since its inception. SimGrid has also been the object of many invalidation and validation studies [42,43,44,45,46], and its simulation models have been shown to provide compelling advantages over other simulation frameworks in terms of both accuracy and scalability [33]. Finally, most SimGrid simulations can be executed in minutes on a standard laptop computer, making it possible to perform large numbers of simulations quickly with minimal compute resource expenses.…”