Multi-site cooperative data stream analysis

Abstract

System S is a large-scale distributed streaming data analysis environment. Ultimately, we envision that there will be multiple sites running the System S software, each with their own administration and goals. However, cooperation between these sites can frequently be of mutual benefit. We are designing the framework to support numerous sites that can work both independently and in cooperative fashions, with a variety of interaction models such as peer-to-peer or federated. Depending on the degree of site autonomy and the relationships between any given pair of sites, the sites would be able to share data, perform processing on behalf of one another, or even take over tasks of a failed site. Interoperability is complicated by the degree of site autonomy as well as differences in execution environments and security policies. This paper surveys System S, describes its models for site interoperability, and discusses particular design issues such as site failover and heterogeneity.

Resource discovery in federated systems with voluntary sharing

“…A representative example is the System S stream processing system [7,4], which allows multiple stream processing sites to collaborate in data stream analysis. Each of these sites consists of many machines owned and operated by one organization.…”

“…System S [7] is an example where multiple stream processing sites, each owned and managed by a different organization, collaborate in complex processing tasks that are difficult to accomplish by any single site. In general, a federated system is composed of loosely coupled autonomous systems that are willing to share their resources for mutual benefits.…”

A Hybrid Approach to High Availability in Stream Processing Systems

“…A processing element (PE) is a software module that takes certain data input, processes it and generates data output for other PEs to consume. To maximize the system utilization, a machine is often shared among different jobs and may host partitions of different jobs concurrently [2], [7], [5], [21].…”

Self healing in System-S

“…Being able to analyze data as it streams rather than storing and using data mining techniques offers the promise of more timely analysis as well as allowing more data to be processed with fewer resources. In this paper, we evaluate some of the autonomic self-healing capabilities of a stream processing system we are prototyping called System-S [1,10,15].…”