White-box testing of big data analytics with complex user-defined functions

Abstract

Data-intensive scalable computing (DISC) systems such as Google's MapReduce, Apache Hadoop, and Apache Spark are being leveraged to process massive quantities of data in the cloud. Modern DISC applications pose new challenges in exhaustive, automatic testing because they consist of dataflow operators, and complex user-defined functions (UDF) are prevalent unlike SQL queries. We design a new white-box testing approach, called BigTest to reason about the internal semantics of UDFs in tandem with the equivalence classes created by each dataflow and relational operator. Our evaluation shows that, despite ultra-large scale input data size, real world DISC applications are often significantly skewed and inadequate in terms of test coverage, leaving 34% of Joint Dataflow and UDF (JDU) paths untested. BigTest shows the potential to minimize data size for local testing by 10 5 to 10 8 orders of magnitude while revealing 2X more manually-injected faults than the previous approach. Our experiment shows that only few of the data records (order of tens) are actually required to achieve the same JDU coverage as the entire production data. The reduction in test data also provides CPU time saving of 194X on average, demonstrating that interactive and fast local testing is feasible for big data analytics, obviating the need to test applications on huge production data. CCS CONCEPTS • Software and its engineering → Cloud computing; Software testing and debugging; • Information systems → MapReduce-based systems.

SPOT: Testing Stream Processing Programs with Symbolic Execution and Stream Synthesizing

“…JDU path coverage. JDU (Joint Dataflow and UDF) path coverage is introduced by Gulzar et al [13], which consider the paths thoroughly along with the operators and the internal paths in their UDFs. Their experimental results demonstrate that JDU path coverage is directly related to improvement in fault detection.…”

“…As discussed in [13], the common industry practice to test big data programs is running locally with randomly sampled data. An empirical study presented by Vianna et al [11] demonstrates that difficulties in generating test data are one of the most frequent problems when designing DSP programs.…”

“…However, real data, including historical and real-time production data, may be privacy-sensitive and hinder developers from accessing conveniently. Taking the ultra-large-scale real data as the test data or randomly generating synthetic data is inefficient as the significant skewness exists for real-world data, leading to low coverage [13]. In this paper, we present two goals that one DSP test approach should reach.…”

“…Symbolic execution [14] is one of the most promising techniques used for generating test cases automatically with high coverage guarantees, working as the foundation of many popular testing tools: Java PathFinder [15] (JPF for short), CUTE [16], and jCUTE [17], KLEE [18], et cetera. However, these solutions and tools are not directly appliable to DSP programs because of the large scale of their frameworks [13]. To tackle this, there are some symbolic execution-based test approaches specifically for big data programs [13,19,20].…”

“…However, these solutions and tools are not directly appliable to DSP programs because of the large scale of their frameworks [13]. To tackle this, there are some symbolic execution-based test approaches specifically for big data programs [13,19,20]. These methods mainly include three steps: (1) make a logical abstraction for dataflow operators, (2) conduct symbolic execution on the UDFs individually and get PCs (i.e., Path Condition), (3) join individual PCs to synthesize the holistic PCs.…”