Stream Processing in Production-to-Business Software

Abstract

In order to support continuous queries over data streams, a plethora of suitable techniques as well as prototypes have been developed and evaluated in recent years. In particular, it is of utmost importance to confirm their necessity and feasibility in real-world applications. For that reason, we have successfully coupled our infrastructure for data stream processing (PIPES) with an industrial Production-to-Business software (i-Plant) dedicated to highly automated manufacturing processes. PIPES: Stream Processing PIPES (Public Infrastructure for Processing and Exploring Streams)[3] is a powerful toolkit providing the essential components to build and execute continuous queries over data streams. The library character of PIPES substantially facilitates the construction of a data stream management system (DSMS) tailored to the particular requirements of an application domain. Our demonstration validates this approach by showing up the intuitive and fruitful composition of PIPES and i-Plant.PIPES relies on a sound semantics [4] which is compatible with the Continuous Query Language (CQL). Its core is a push-based, time-interval operator algebra. Continuous queries are implemented as a directed acyclic operator graph which enables PIPES to gain from subquery sharing.In addition, PIPES provides flexible frameworks for the runtime components, e. g., scheduler, memory manager, and query optimizer. Each component is parameterized by a strategy in order to guarantee runtime adaptivity. Appropriate strategies are usually based on a cost model [1] which incorporates continuously gathered runtime metadata such as stream rates, memory usage, and operator costs. As more complex metadata, PIPES offers online, resource-adaptive estimations of data distributions based on complex statistical methods, like wavelet-based density estimators [2].

Dynamic Metadata Management for Scalable Stream Processing Systems

“…The dynamic metadata management framework presented in this paper is fully implemented as an integral part of the stream processing infrastructure PIPES [17,8]. PIPES is not only made for programmers who want to set up a scalable stream processing system, but also supports data stream research due to its extensible and highly generic design, which allows to implement and evaluate new techniques easily.…”

“…As this paper summarizes our experiences with realizing dynamic metadata management in our stream processing framework PIPES [17,8], we finally spent some words on implementation issues.…”

Toward Simulation-Based Optimization in Data Stream Management Systems

“…We have implemented this novel optimizer architecture on top of our data stream infrastructure PIPES [3], [4], which is an integral part of our Java library XXL [5]. For illustration purposes, Figure 1 provides an overview of the general architecture, its essential modules, and their interaction.…”

Place: A Distributed Spatio-Temporal Data Stream Management System for Moving Objects

“…Examples of the prototypes include TelegraphCQ [8], NiagaraCQ [10], PSoup [9], STREAM [4,21], Aurora [1], NILE [13], PIPES [7], and CAPE [24]. One common characteristic of the above systems is that moving objects and continuous queries are processed in a centralized fashion.…”