Failure Recovery in Cooperative Data Stream Analysis

Rong, Bin; Douglis, Fred; Liu, Zhen; Xi, Chenguang

doi:10.1109/ares.2007.87

Cited by 2 publications

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

CLASP: Collaborating, Autonomous Stream Processing Systems

Branson

Douglis

Fawcett

et al. 2007

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. There are currently a number of streaming data analysis systems in research or commercial operation. These systems are generally large-scale distributed systems, but each system operates in isolation, under the control of one administrative authority. We are developing middleware that permits autonomous or semi-autonomous streaming analysis systems (called "sites") to interoperate, providing them opportunities for data access, performance improvements, and reliability far exceeding that available in a single system. Unique characteristics of our system include an architecture for the management of multiple cooperation paradigms depending on the degree of trust and dependencies among the participating sites; a multisite planner that converts user-specified declarative queries into specifications of distributed jobs; and a mechanism for automatic recovery of site failures by redispatching failed pieces of a distributed job. We evaluate our architecture via experiments on a running prototype, and the results demonstrate the advantages of multisite cooperation: collaborative jobs that share resources, even across only a few sites, can produce results 50% faster than independent execution, and jobs on failed sites can be recovered within a few seconds.

show abstract

CLASP: Collaborating, Autonomous Stream Processing Systems

Branson

Douglis

Fawcett

et al. 2007

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

A Hybrid Approach to High Availability in Stream Processing Systems

Zhang

et al. 2010

2010 IEEE 30th International Conference on Distributed Computing Systems

View full text Add to dashboard Cite

Abstract-Stream processing is widely used by today's applications such as financial data analysis and disaster response. In distributed stream processing systems, machine fail-stop events are handled by either active standby or passive standby. However, existing HA schemes have not sufficiently addressed the situation when a machine becomes temporarily unavailable due to data rate spikes, intensive analysis or job sharing, which happens frequently but lasts for short time. It is not clear how well active and passive standby fare against such transient unavailability. In this paper, we first critically examine the suitability of active and passive standby against transient unavailability in a real testbed environment. We find that both approaches have advantages and drawbacks, but neither is ideal to provide fast recovery at low overhead as required to handle transient unavailability. Based on the insights gained, we propose a novel hybrid HA method that switches between active and passive standby modes depending on the occurrence of failure events. It presents a desirable tradeoff that is different from existing HA approaches: low overhead during normal conditions and fast recovery upon transient or permanent failure events. We have implemented our hybrid method and compared it with existing HA designs with comprehensive evaluation. The results show that our hybrid method can reduce two-thirds of the recovery time compared to passive standby and 80% message overhead compared to active standby, allowing applications to enjoy uninterrupted processing without paying a high premium.

show abstract

Failure Recovery in Cooperative Data Stream Analysis

Cited by 2 publications

References 16 publications

CLASP: Collaborating, Autonomous Stream Processing Systems

CLASP: Collaborating, Autonomous Stream Processing Systems

A Hybrid Approach to High Availability in Stream Processing Systems

Contact Info

Product

Resources

About