Detection and analysis of resource usage anomalies in large distributed systems through multi‐scale visualization

Schnorr, Lucas Mello; Legrand, Arnaud; Vincent, Jean-Marc

doi:10.1002/cpe.1885

Cited by 12 publications

(13 citation statements)

References 40 publications

(61 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In addition to the MSG API, SimGrid provides several abstractions such as the TRACE mechanisms that enable to perform fine post-mortem analysis of the users' simulations with some visualization tools like the Triva/Viva/PajeNG software suite [5].…”

Section: Simgrid Is Composed Of Three Different Layersmentioning

confidence: 99%

SimGrid VM: Virtual Machine Support for a Simulation Framework of Distributed Systems

Hirofuchi

Lèbre

Pouilloux

2018

IEEE Trans. Cloud Comput.

View full text Add to dashboard Cite

International audienceAs real systems become larger and more complex, the use of simulator frameworks grows in our research community. By leveraging them, users can focus on the major aspects of their algorithm, run in-siclo experiments (i.e., simulations), and thoroughly analyze results, even for a large-scale environment without facing the complexity of conducting in-vivo studies (i.e., on real testbeds). Since nowadays the virtual machine (VM) technology has become a fundamental building block of distributed computing environments, in particular in cloud infrastructures, our community needs a full-fledged simulation framework that enables us to investigate large-scale virtualized environments through accurate simulations. To be adopted, such a framework should provide easy-to-use APIs as well as accurate simulation results. In this paper, we present a highly-scalable and versatile simulation framework supporting VM environments. By leveraging SimGrid, a widely-used open-source simulation toolkit, our simulation framework allows users to launch hundreds of thousands of VMs on their simulation programs and control VMs in the same manner as in the real world (e.g., suspend/resume and migrate). Users can execute computation and communication tasks on physical machines (PMs) and VMs through the same SimGrid API, which will provide a seamless migration path to IaaS simulations for hundreds of SimGrid users. Moreover, SimGrid VM includes a live migration model implementing the precopy migration algorithm. This model correctly calculates the migration time as well as the migration traffic, taking account of resource contention caused by other computations and data exchanges within the whole system. This allows user to obtain accurate results of dynamic virtualized systems. We confirmed accuracy of both the VM and the live migration models by conducting several micro-benchmarks under various conditions. Finally, we conclude the article by presenting a first use-case of one consolidation algorithm dealing with a significant number of VMs/PMs. In addition to confirming the accuracy and scalability of our framework, this first scenario illustrates the main interest of SimGrid VM: investigating through in-siclo experiments pros/cons of new algorithms in order to limit expensive in-vivo experiments only to the most promising ones

show abstract

Section: Simgrid Is Composed Of Three Different Layersmentioning

confidence: 99%

SimGrid VM: Virtual Machine Support for a Simulation Framework of Distributed Systems

Hirofuchi

Lèbre

Pouilloux

2018

IEEE Trans. Cloud Comput.

View full text Add to dashboard Cite

show abstract

“…Our new experimental scenarios have larger errors, making "bad cases" easier to identify. To understand the root causes of the errors we use a multi-scale, interactive and exploratory visualization tool called Triva [Schnorr et al 2011;Schnorr et al 2010;Triva 2011]. We select some of the scenarios that have large maximum error and run them until completion of all flows so that we can compare flow completion times and not only instantaneous bandwidth shares.…”

Section: Reverse Traffic Effectsmentioning

confidence: 99%

On the validity of flow-level tcp network models for grid and cloud simulations

Velho

Schnorr

Casanova

et al. 2013

ACM Trans. Model. Comput. Simul.

Self Cite

View full text Add to dashboard Cite

Researchers in the area of grid/cloud computing perform many of their experiments using simulations that must capture network behavior. In this context, packet-level simulations, which are widely used to study network protocols, are too costly given the typical large scales of simulated systems and applications. An alternative is to implement network simulations with less costly flow-level models. Several flow-level models have been proposed and implemented in grid/cloud simulators. Surprisingly, published validations of these models, if any, consist of verifications for only a few simple cases. Consequently, even when they have been used to obtain published results, the ability of these simulators to produce scientifically meaningful results is in doubt. This work evaluates these state-of-the-art flow-level network models of TCP communication via comparison to packet-level simulation. While it is straightforward to show cases in which previously proposed models lead to good results, instead we follow the critical method, which places model refutation at the center of the scientific activity, and we systematically seek cases that lead to invalid results. Careful analysis of these cases reveal fundamental flaws and also suggest improvements. One contribution of this work is that these improvements lead to a new model that, while far from being perfect, improves upon all previously proposed models in the context of simulation of grids or clouds. A more important contribution, perhaps, is provided by the pitfalls and unexpected behaviors encountered in this work, leading to a number of enlightening lessons. In particular, this work shows that model validation cannot be achieved solely by exhibiting (possibly many) "good cases." Confidence in the quality of a model can only be strengthened through an invalidation approach that attempts to prove the model wrong.

show abstract

“…Visualization Scalability : An embedded system execution trace usually contains too much information to be entirely represented on the screen. This could result in cluttered drawings, non-exact proportions or uncontrolled visual aggregation [11]. Furthermore, dierent information needs to be represented dierently.…”

Section: Storage Of Big Traces : Execution Traces Of Embedded Systemsmentioning

confidence: 99%

Trace Management and Analysis for Embedded Systems

Pagano

Dosimont

Huard³

et al. 2013

2013 IEEE 7th International Symposium on Embedded Multicore Socs

Self Cite

View full text Add to dashboard Cite

International audienceThe growing complexity of embedded systems makes their behavior analysis a challenging task. In this context, tracing appears to be a promising solution as it provides relevant information about the system execution. However, trace management and analysis are hindered by the diversity of trace formats, the incompatibility of trace analysis methods, the problem of trace size and its storage as well as by the lack of visualization scalability. In this article we present FrameSoC, a new trace management infrastructure that solves these issues. It proposes generic solutions for trace storage and defines interfaces and plug in mechanisms for integrating diverse analysis tools. We illustrate the benefit of FrameSoC with a case study of a visualization module that provides representation scalability for large traces by using an aggregation algorithm

show abstract

Detection and analysis of resource usage anomalies in large distributed systems through multi‐scale visualization

Cited by 12 publications

References 40 publications

SimGrid VM: Virtual Machine Support for a Simulation Framework of Distributed Systems

SimGrid VM: Virtual Machine Support for a Simulation Framework of Distributed Systems

On the validity of flow-level tcp network models for grid and cloud simulations

Trace Management and Analysis for Embedded Systems

Contact Info

Product

Resources

About