An Exploratory Study of How Specialists Deal with Testing in Data Stream Processing Applications

Abstract

Background] Nowadays, there is a massive growth of data volume and speed in many types of systems. It introduces new needs for infrastructure and applications that have to handle streams of data with low latency and high throughput. Testing applications that process such data streams has become a significant challenge for engineers. Companies are adopting different approaches to dealing with this issue. Some have developed their own solutions for testing, while others have adopted a combination of existing testing techniques. There is no consensus about how or in which contexts such solutions can be implemented.[Aims] To the best of our knowledge, there is no consolidated literature on that topic. The present paper is an attempt to fill this gap by conducting an exploratory study with practitioners. [Method] We used qualitative methods in this research, in particular interviews and survey. We interviewed 12 professionals who work in projects related to data streams, and also administered a questionnaire with other 105 professionals. The interviews went through a transcription and coding process, and the questionnaires were analysed to reinforce findings.[Results] This study presents current practices around software testing in data stream processing applications. These practices involve methodologies, techniques, and tools. [Conclusions] Our main contribution is a compendium of alternatives for many of the challenges that arise when testing streaming applications from a state-of-the-practice perspective.

A qualitative study of Machine Learning practices and engineering challenges in Earth Observation

“…Conducting semi-structured expert interviews is a wellestablished procedure, originally from psychology and social science, and proven for the exploratory examination of requirements in both the contexts of software engineering [20,14] and ML [3,7]. Compared to fully structured interviews there is some flexibility regarding the course of a conversation, which allows the experts to express their very own perception without being primed by too explicit questions.…”

“…Within the ML community, there is a broad understanding of what ML workflows look like in reality. Numerous practical blog posts and textbooks aim to make these abstract conceptions tangible (e. g. [22,20,5,1]). Of course, there are differences in the variety of depictions.…”

SPOT: Testing Stream Processing Programs with Symbolic Execution and Stream Synthesizing

“…Test activities are essential for DSP programs' quality assurance and to improve dependability. The importance is fully emphasized in a recent empirical study [11]. Software testing is a significant cost during the software development life cycle, and there exists a strong demand to design an efficient test solution for the expense-reducing purpose.…”

“…As discussed in [13], the common industry practice to test big data programs is running locally with randomly sampled data. An empirical study presented by Vianna et al [11] demonstrates that difficulties in generating test data are one of the most frequent problems when designing DSP programs. In practice, test data for DSP programs mainly comes from three sources [11], i.e., replaying historical data, mirroring real-time production data, and generating synthetic data randomly.…”

“…An empirical study presented by Vianna et al [11] demonstrates that difficulties in generating test data are one of the most frequent problems when designing DSP programs. In practice, test data for DSP programs mainly comes from three sources [11], i.e., replaying historical data, mirroring real-time production data, and generating synthetic data randomly. However, real data, including historical and real-time production data, may be privacy-sensitive and hinder developers from accessing conveniently.…”