A Model for Space-Correlated Failures in Large-Scale Distributed Systems

Gallet, Matthieu; Yigitbasi, Nezih; Javadi, Bahman; Kondo, Derrick; Iosup, Alexandru; Epema, Dick

doi:10.1007/978-3-642-15277-1_10

Cited by 35 publications

(48 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The FTA traces have been instrumental in the development of various failure models [46,55,56]. In a recent study [46], we analyzed and model the time-varying behavior of failures in large-scale distributed systems.…”

Section: Discussion: On the Current And Future Use Of The Ftamentioning

confidence: 99%

The Failure Trace Archive: Enabling the comparison of failure measurements and models of distributed systems

Javadi

Kondo

Iosup

et al. 2013

Journal of Parallel and Distributed Computing

Self Cite

View full text Add to dashboard Cite

With the increasing presence, scale, and complexity of distributed systems, resource failures are becoming an important and practical topic of computer science research. While numerous failure models and failure-aware algorithms exist, their comparison has been hampered by the lack of public failure data sets and data processing tools. To facilitate the design, validation, and comparison of fault-tolerant models and algorithms, we have created the Failure Trace Archive (FTA)-an online, public repository of failure traces collected from diverse parallel and distributed systems. In this work, we first describe the design of the archive, in particular of the standard FTA data format, and the design of a toolbox that facilitates automated analysis of trace data sets. We also discuss the use of the FTA for various current and future purposes. Second, after applying the toolbox to nine failure traces collected from distributed systems used in various application domains (e.g., HPC, Internet operation, and various online applications), we present a comparative analysis of failures in various distributed systems. underline the need for public availability of trace data from different distributed systems. Last, we show how different interpretations of the meaning of failure data can result in different conclusions for failure modeling and job scheduling in distributed systems. Our results for different interpretations show evidence that there may be a need for further revisiting existing failureaware algorithms, when applied for general rather than for domain-specific distributed systems.

show abstract

Section: Discussion: On the Current And Future Use Of The Ftamentioning

confidence: 99%

The Failure Trace Archive: Enabling the comparison of failure measurements and models of distributed systems

Javadi

Kondo

Iosup

et al. 2013

Journal of Parallel and Distributed Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, they pose more fundamental challenges for problem determination. Problem determination in traditional distributed systems has always focused on one application at a time [8]. Collocation faults imply that the model for normal behavior of an application needs to be learned in the context of other collocated applications.…”

Section: B End-to-end Problem Determination In a Cloud Ecosystemmentioning

confidence: 99%

“…We configure a cluster of 8 Hadoop Sort benchmark is run on 5 GB of data. We correlate CPU, memory and disk utilization across the VMs and report their average values in Figure 7.…”

Section: ) Stability Of Correlation With Change In Vm Configurationmentioning

confidence: 99%

“…A related study [7] on 3 years worth forum messages concerning the problems faced by end users of utility clouds shows that virtualization related issues contribute to around 20% of the total problems experienced. The existence of public repository of failure traces [8] across diverse distributed systems like Skype, Microsoft and Planetlab, further demonstrates the prevalence and need for taming the faults for successful operation.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CloudPD: Problem determination and diagnosis in shared dynamic clouds

Sharma

Jayachandran

Verma

et al. 2013

2013 43rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)

View full text Add to dashboard Cite

Abstract-In this work, we address problem determination in virtualized clouds. We show that high dynamism, resource sharing, frequent reconfiguration, high propensity to faults and automated management introduce significant new challenges towards fault diagnosis in clouds. Towards this, we propose CloudPD, a fault management framework for clouds. CloudPD leverages (i) a canonical representation of the operating environment to quantify the impact of sharing; (ii) an online learning process to tackle dynamism; (iii) a correlation-based performance models for higher detection accuracy; and (iv) an integrated end-to-end feedback loop to synergize with a cloud management ecosystem. Using a prototype implementation with cloud representative batch and transactional workloads like Hadoop, Olio and RUBiS, it is shown that CloudPD detects and diagnoses faults with low false positives (< 16%) and high accuracy of 88%, 83% and 83%, respectively. In an enterprise trace-based case study, CloudPD diagnosed anomalies within 30 seconds and with an accuracy of 77%, demonstrating its effectiveness in real-life operations.

show abstract

“…There are other papers that discuss correlated failures in various contexts ( [12] and [13]) other than communication networks. There are also models for temporally correlated failures [14].…”

Section: Introductionmentioning

confidence: 99%

Modeling Stochastic Correlated Failures and their Effects on Network Reliability

Rahnamay-Naeini

Pezoa

Azar

et al. 2011

2011 Proceedings of 20th International Conference on Computer Communications and Networks (ICCCN)

View full text Add to dashboard Cite

Abstract-The physical infrastructure of communication networks is vulnerable to spatially correlated failures arising from various physical stresses such as natural disasters (earthquakes and hurricanes) as well as malicious coordinated attacks using weapons of mass destruction. Some disaster events such as earthquakes and terrorist attacks may occur in more than one location in a short period of time. Hence multiple sets of correlated link failures may occur if more events occurred before the previous set of failed links were repaired. Here, the statistical properties of induced-failure patterns depend upon the spatial interaction among stress centers (e.g., interaction among earthquake or attack locations). This paper presents a stochastic model, based on spatial point processes, for representing stress centers on the geographical plane in order to facilitate the modeling of spatially inhomogeneous and correlated link failures in communication networks. This model is then used to further generate scenarios with inhibition or clustering between stress centers, which enables detailed assessment of vulnerabilities of the network to the level of inhomogeneity and spatial correlation in the stress-event centers. Detailed simulation results are presented to compare network reliability for various scenarios of link failures and to find geographically vulnerable areas of a network as well as worstcase scenarios of stress-events. Overall, this effort will provide some critical knowledge and simulation capabilities for other focus areas of research in network reliability and survivability.

show abstract

A Model for Space-Correlated Failures in Large-Scale Distributed Systems

Cited by 35 publications

References 19 publications

The Failure Trace Archive: Enabling the comparison of failure measurements and models of distributed systems

The Failure Trace Archive: Enabling the comparison of failure measurements and models of distributed systems

CloudPD: Problem determination and diagnosis in shared dynamic clouds

Modeling Stochastic Correlated Failures and their Effects on Network Reliability

Contact Info

Product

Resources

About