CLASP: Collaborating, Autonomous Stream Processing Systems

Abstract

Abstract. There are currently a number of streaming data analysis systems in research or commercial operation. These systems are generally large-scale distributed systems, but each system operates in isolation, under the control of one administrative authority. We are developing middleware that permits autonomous or semi-autonomous streaming analysis systems (called "sites") to interoperate, providing them opportunities for data access, performance improvements, and reliability far exceeding that available in a single system. Unique characteristics of our system include an architecture for the management of multiple cooperation paradigms depending on the degree of trust and dependencies among the participating sites; a multisite planner that converts user-specified declarative queries into specifications of distributed jobs; and a mechanism for automatic recovery of site failures by redispatching failed pieces of a distributed job. We evaluate our architecture via experiments on a running prototype, and the results demonstrate the advantages of multisite cooperation: collaborative jobs that share resources, even across only a few sites, can produce results 50% faster than independent execution, and jobs on failed sites can be recovered within a few seconds.

A Hybrid Approach to High Availability in Stream Processing Systems

“…It reuses some of the components in our previous CLASP [5] work. It can support four different HA modes for each subjob: No HA (NONE), where a single copy is deployed and no action is taken when failures happen; Active Standby (AS), where two copies are deployed and both process data and send output downstream; Passive Standby (PS), where another machine runs the secondary copy upon failures; and Hybrid, where another machine has a pre-deployed but suspended secondary copy.…”

“…A processing element (PE) is a software module that takes certain data input, processes it and generates data output for other PEs to consume. To maximize the system utilization, a machine is often shared among different jobs and may host partitions of different jobs concurrently [2], [7], [5], [21].…”

Embedding intelligent decision making within complex dynamic environments

“…Given these factors, optimal utilization of these systems is an imperative as is the sharing of vast amounts of data/information to support scientific research efforts of local and global importance. A new approach for collaboration across stream processing systems has been reported as an enhancement to System S and is known as CLASP (Collaborating, Autonomous Stream Processing) (Branson et al 2007). CLASP provides several benefits including improved sharing of raw data streams and derived data, thereby improving the breadth of analyses.…”

Strategic agents for multi-resource negotiation

“…We consider both assumptions are realistic in practice. For example, consider the streaming processing system CLASP [7], each resource provider (consumer) always posts its resource supply (requirement). Further, the distribution of trading partners' reserve prices can be learned as a result of repeated interaction with agents in the marketplace.…”

“…As an example, consider the negotiation management component [1] for Collaborating, Autonomous Stream Processing systems (CLASP) [7], which has been designed and prototyped in the context of System S project [19] within IBM Research to enable sophisticated stream processing. There are multiple sites running the System S software, each with their own administration and goals.…”