Dependable Cloud Resources with Guardian

Abusalah, Bara; Schatzlein, Derek; Stephen, Julian James; Ardekani, Masoud Saeida; Eugster, Patrick

doi:10.1109/icdcs.2017.158

Cited by 3 publications

(6 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Guardian improves completion time by an average of 68% in the presence of failures, while maintaining a small increase in overhead compared to no replication in unsaturated environments (about 6% overhead in the highest replication degree). Part of this work is based on lessons learned from our previous work [15]. For instance, we improve how result verification is done in [15] by introducing Hybrid Verify in Section 4.1.3.…”

Section: Contributions and Roadmapmentioning

confidence: 99%

“…Part of this work is based on lessons learned from our previous work [15]. For instance, we improve how result verification is done in [15] by introducing Hybrid Verify in Section 4.1.3. Hybrid Verify takes advantage of the benefits of the verification forms mentioned in [15] while avoiding their drawbacks.…”

Section: Contributions and Roadmapmentioning

confidence: 99%

“…For instance, we improve how result verification is done in [15] by introducing Hybrid Verify in Section 4.1.3. Hybrid Verify takes advantage of the benefits of the verification forms mentioned in [15] while avoiding their drawbacks. Another motivation for expanding our work in [15] is to make our FT approach universal to multiple RMSs.…”

Section: Contributions and Roadmapmentioning

confidence: 99%

“…Hybrid Verify takes advantage of the benefits of the verification forms mentioned in [15] while avoiding their drawbacks. Another motivation for expanding our work in [15] is to make our FT approach universal to multiple RMSs. This is supported by extending Guardian to RMSs other than the one used in [15] (YARN).…”

Section: Contributions and Roadmapmentioning

confidence: 99%

“…Another motivation for expanding our work in [15] is to make our FT approach universal to multiple RMSs. This is supported by extending Guardian to RMSs other than the one used in [15] (YARN). Furthermore, we show that our replication approach to achieve FT at the RMS level is still effective even with the challenging resource offer system used in Mesos.…”

Section: Contributions and Roadmapmentioning

confidence: 99%

See 4 more Smart Citations

Multi-Framework Reliability Approach

Abusalah

Schatzlein

Stephen

et al. 2022

IEEE Trans. Cloud Comput.

Self Cite

View full text Add to dashboard Cite

Despite advances in making datacenters dependable, failures still happen. This is particularly onerous for long-running "big data" applications, where partial failures can lead to significant losses and lengthy recomputations. Big data processing frameworks like Hadoop MapReduce include fault tolerance (FT) mechanisms, but these are commonly targeted at specific system/failure models, and are often redundant between frameworks. This paper proposes the paradigm of dependable resources: big data processing frameworks are typically built on top of resource management systems (RMSs), and proposing FT support at the level of such an RMS yields generic FT mechanisms, which can be provided with low overhead by leveraging constraints on resources. We demonstrate our concepts through Guardian, a robust RMS based on Mesos and YARN. Guardian allows frameworks to run their applications with individually configurable FT granularity and degree, with only minor changes to their implementation. We demonstrate the benefits of our approach by evaluating Hadoop, Tez, Spark and Pig on a prototype of Guardian running on Amazon-EC2, improving completion time by around 68% in the presence of failures, while maintaining around 6% overhead.

show abstract

Section: Contributions and Roadmapmentioning

confidence: 99%

Section: Contributions and Roadmapmentioning

confidence: 99%

Section: Contributions and Roadmapmentioning

confidence: 99%

Section: Contributions and Roadmapmentioning

confidence: 99%

Section: Contributions and Roadmapmentioning

confidence: 99%

See 3 more Smart Citations

Multi-Framework Reliability Approach

Abusalah

Schatzlein

Stephen

et al. 2022

IEEE Trans. Cloud Comput.

Self Cite

View full text Add to dashboard Cite

show abstract

Interminable Flows: A Generic, Joint, Customizable Resiliency Model for Big-Data Streaming Platforms

et al. 2023

Self Cite

View full text Add to dashboard Cite

The examiner of cloud computing systems in the last few years observes that there is a trend of the emergence of a new Big Data framework almost every year. Since Hadoop was developed in 2007, new frameworks followed it such as Spark, Storm, Heron, Apex, Flink, Samza, Kafka. . . etc. Each framework is developed in a certain way to target and achieve certain objectives better than other frameworks do. However, there are few common functionalities and aspects that are shared between these frameworks. One vital aspect all these frameworks strive to achieve is better reliability and faster recovery time in case of failures. This is particularly crucial for streaming systems (compared to batch processing systems) where events are processed and monitored online in real time, and any delay in data delivery will cause a major inconvenience to the users. Another observation is that some reliability implementations are redundant between different frameworks. Encapsulating these implementations into one layer and make it shared between different applications will benefit more than one framework without the burden of re-implementing the same reliability approach in each single framework. These observations motivated us to present Warden, a generic, multiframework, flexible, customizable, low overhead protocol to ensure the resiliency of streaming applications running on streaming Big Data frameworks. Most reliability protocols carry out one rigid fault tolerance approach targeted towards one system at a time. It is more challenging to provide a reliability approach that is pluggable in multiple Big Data frameworks at a time and can achieve low overheads comparable with single targeted framework approaches, yet is flexible and customizable by its users to make it tailored towards their objectives. The genericity is attained by providing an interface that can be used in different applications from different frameworks. The low overhead is achieved by providing faster application finish times with and without failures. The customizability is fulfilled by providing the users the options to choose between two delivery semantics (Exactly Once / At Most Once) combined with two fault tolerance guarantees (Crash Failures / Byzantine Failures). To the best of our knowledge, such approach was never tried on multiple streaming frameworks before. We built a prototype of Warden on Flink and Samza (with Kafka) streaming frameworks. Our evaluations highlight the effectiveness of our approach in the presence of failures and without failures compared to other fault tolerance techniques (such as checkpointing).

show abstract