Testing properties of dataflow program operators

IEEE Trans. Parallel Distrib. Syst.

et al. 2014

Self Cite

205

201

This article addresses the profitability problem associated with auto-parallelization of general-purpose distributed data stream processing applications. Auto-parallelization involves locating regions in the application's data flow graph that can be replicated at run-time to apply data partitioning, in order to achieve scale. In order to make auto-parallelization effective in practice, the profitability question needs to be answered: How many parallel channels provide the best throughput? The answer to this question changes depending on the workload dynamics and resource availability at run-time. In this article, we propose an elastic auto-parallelization solution that can dynamically adjust the number of channels used to achieve high throughput without unnecessarily wasting resources. Most importantly, our solution can handle partitioned stateful operators via run-time state migration, which is fully transparent to the application developers. We provide an implementation and evaluation of the system on an industrial-strength data stream processing platform to validate our solution.

Section: Auto-parallelizationmentioning

confidence: 99%

Elastic Scaling for Data Stream Processing

Gedik

Schneider

IEEE Trans. Parallel Distrib. Syst.

et al. 2014

Self Cite

205

201

“…A second use case for our testing framework, explored initially in [23], involves the testing of Map and Reduce functions by engineers who create them. In this use case, engineers can utilize test cases that they create, or test cases generated by approaches such as the one we utilize in our study, to assess whether their operators possess necessary properties for the given situations in which they are employed.…”

Section: Discussionmentioning

confidence: 99%

“…Finally, we discuss test input generation techniques. In related work [23], we built a similar testing framework for a different domain (streaming analytics using SPL [12] instead of batch analytics using MapReduce [7]). Figure 2 shows our testing framework.…”

Section: Dynamic Analysismentioning

confidence: 99%

See 1 more Smart Citation

Semantic characterization of MapReduce workloads

2013 IEEE International Symposium on Workload Characterization (IISWC)

Rothermel

2013

Self Cite

Abstract-MapReduce is a platform for analyzing large amounts of data on clusters of commodity machines. MapReduce is popular, in part thanks to its apparent simplicity. However, there are unstated requirements for the semantics of MapReduce applications that can affect their correctness and performance. MapReduce implementations do not check whether user code satisfies these requirements, leading to time-consuming debugging sessions, performance problems, and, worst of all, silently corrupt results. This paper makes these requirements explicit, framing them as semantic properties and assumed outcomes. It describes a black-box approach for testing for these properties, and uses the approach to characterize the semantics of 23 non-trivial MapReduce workloads. Surprisingly, we found that for most requirements, there is at least one workload that violates it. This means that MapReduce may be simple to use, but it is not as simple to use correctly. Based on our results, we provide insights to users on how to write higher-quality MapReduce code, and insights to system and language designers on ways to make their platforms more robust.

“…Our approach to partitioning guarantees partition isolation [44], i.e., the combination of two properties:…”

Section: Semanticsmentioning

confidence: 99%

Spreadsheets for stream processing with unbounded windows and partitions

Proceedings of the 10th ACM International Conference on Distributed and Event-Based Systems

Rabbah

Suter

et al. 2016

Self Cite

Stream processing is a computational paradigm that allows the analysis of live data streams as they are produced. This paper describes a programming model, based on enhancements to spreadsheets, that enables users with limited programming experience to participate directly in the development of complex streaming applications. The programming model augments a conventional spreadsheet with streaming features that permit operating over unbounded data sets despite the finite interface provided by the spreadsheet. The new constructs include time-based windows and partitioning. We introduce a spreadsheet compiler that generates C++ code to achieve integration with existing stream processing systems. Our experimental study illustrates the expressivity of the new features and finds that our implementation is between 8x slower and 2x faster than hand-written stream programs.