On the validity of flow-level tcp network models for grid and cloud simulations

Abstract

Researchers in the area of grid/cloud computing perform many of their experiments using simulations that must capture network behavior. In this context, packet-level simulations, which are widely used to study network protocols, are too costly given the typical large scales of simulated systems and applications. An alternative is to implement network simulations with less costly flow-level models. Several flow-level models have been proposed and implemented in grid/cloud simulators. Surprisingly, published validations of these models, if any, consist of verifications for only a few simple cases. Consequently, even when they have been used to obtain published results, the ability of these simulators to produce scientifically meaningful results is in doubt. This work evaluates these state-of-the-art flow-level network models of TCP communication via comparison to packet-level simulation. While it is straightforward to show cases in which previously proposed models lead to good results, instead we follow the critical method, which places model refutation at the center of the scientific activity, and we systematically seek cases that lead to invalid results. Careful analysis of these cases reveal fundamental flaws and also suggest improvements. One contribution of this work is that these improvements lead to a new model that, while far from being perfect, improves upon all previously proposed models in the context of simulation of grids or clouds. A more important contribution, perhaps, is provided by the pitfalls and unexpected behaviors encountered in this work, leading to a number of enlightening lessons. In particular, this work shows that model validation cannot be achieved solely by exhibiting (possibly many) "good cases." Confidence in the quality of a model can only be strengthened through an invalidation approach that attempts to prove the model wrong.

Schlouder: A broker for IaaS clouds

“…Item a) has been studied in-depth regarding the network model [11], while the CPU model does not yet capture all the complexity induced for instance by the cache effects. Concerning item b), while we can easily capture this information in our private cloud testbeds, it is usually unknown for public clouds.…”

Adding Storage Simulation Capacities to the SimGrid Toolkit: Concepts, Models, and API

“…We also build tractable models that derive from a thorough characterization of storage resources and take important phenomena such as sharing and contention effects into account. We leverage the SimGrid expertise in network modeling [16] to adapt coarsegrain fluid models to the simulation of storage resources.…”

“…SimGrid relies on a scalable and extensible simulation engine and offers several user APIs, which we detail hereafter. This tool gained a strong expertise in discrete-event simulation, especially in network modeling [16]. While storage is as much important as compute nodes or interconnection networks in many large scale distributed computing infrastructures, SimGrid did not provide any storage abstraction until the proposed extension implemented in its latest stable release (3.11.1).…”

“…The key aspect of this model is that it is general and can be used to simulate key aspects of CPU and network. The validity of this type of models in the context of networking has been thoroughly assessed [16].…”

“…However, this popular wisdom was rebutted for network resources by the flow-level models implemented within the SimGrid toolkit. Not only they can be as accurate as their packet-level contenders, but they are also orders of magnitude faster (see [16] for details). We thus believe that flow-level models is a sound approach to the coarse-grain simulation of storage resources.…”