Tools and strategies for debugging distributed stream processing applications

Abstract

Distributed data stream processing applications are often characterized by data flow graphs consisting of a large number of built‐in and user‐defined operators connected via streams. These flow graphs are typically deployed on a large set of nodes. The data processing is carried out on‐the‐fly, as tuples arrive at possibly very high rates, with minimum latency. It is well known that developing and debugging distributed, multi‐threaded, and asynchronous applications, such as stream processing applications, can be challenging. Thus, without domain‐specific debugging support, developers struggle when debugging distributed applications. In this paper, we describe tools and language support to support debugging distributed stream processing applications. Our key insight is to view debugging of stream processing applications from four different, but related, perspectives. First, debugging the semantics of the application involves verifying the operator‐level composition and inspecting the flows at the logical level. Second, debugging the user‐defined operators involves traditional source‐code debugging, but strongly tied to the stream‐level interactions. Third, debugging the deployment details of the application require understanding the runtime physical layout and configuration of the application. Fourth, debugging the performance of the application requires inspecting various performance metrics (such as communication rates, CPU utilization, etc.) associated with streams, operators, and nodes in the system. In light of this characterization, we developed several tools such as a debugger‐aware compiler and an associated stream debugger, composition and deployment visualizers, and performance visualizers, as well as language support, such as configuration knobs for logging and tracing, deployment configurations such as operator‐to‐process and process‐to‐node mappings, monitoring directives to inspect streams, and special sink adapters to intercept and dump streaming data to files and sockets, to name a few. We describe these tools in the context of Spade—a language for creating distributed stream processing applications, and System S—a distributed stream processing middleware under development at the IBM Watson Research Center. Published in 2009 by John Wiley & Sons, Ltd.

Design principles for developing stream processing applications

“…We continue to evolve the System S platform and its programming language 30, along with its associated integrated development and visualization environment 40, 42, focusing on providing the infrastructure and the abstractions needed by application analysts and developers as they undertake the task of implementing ever more challenging and large‐scale applications, bringing us closer to the vision of processing information via true sense‐and‐respond systems 59.…”

“…start with a few operators and gradually grow the graph, while keeping track of the throughput at each step. Equally important is the use of adequate tooling for application understanding so that optimization decisions are rooted on quantitative information 40, 42.…”

SpamWatcher

“…SPL [15,16] is the programming language of System S. The SPL tooling includes a rapid application development environment, as well as visualization and debugging tools [8,13]. The language can be used to compose parallel and distributed stream processing applications, in the form of operator-based dataflow graphs.…”

From a stream of relational queries to distributed stream processing

“…Spade [20,23] is the programming language for System S. The Spade tooling includes a rapid application development environment, as well as visualization and debugging tools [12,18]. The language can be used to compose parallel and distributed stream processing applications, in the form of operator-based dataflow graphs.…”