Commutativity of Reducers

Chen, Yu‐Fang; Hong, Chih-Duo; Sinha, Nishant; Wang, Bow-Yaw

doi:10.1007/978-3-662-46681-0_9

Cited by 18 publications

(20 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Camargo and Vergilio classify the specific faults of MapReduce , whereas Morán et al classify those caused by the nondeterminism execution. Chen et al propose a formal approach to detect these faults caused by nondeterminism. In contrast, Csallner et al employ symbolic execution to check the program under test.…”

Section: Resultsmentioning

confidence: 99%

Testing MapReduce programs: A systematic mapping study

Morán

Riva

Tuya

2018

J Software Evolu Process

View full text Add to dashboard Cite

Summary Context MapReduce is a processing model used in Big Data to facilitate the analysis of large data under a distributed architecture. Objective The aim of this study is to identify and categorize the state of the art of software testing in MapReduce applications, determining trends and gaps. Method Systematic mapping study to discuss and classify according to international standards 54 relevant studies in relation to reasons for testing, types of testing, quality characteristics, test activities, tools, roles, processes, test levels, and research validations. Results The principal reasons for testing MapReduce applications are performance issues, potential failures, issues related to the data, or to satisfy the agreements with efficient resources. The efforts are focused on performance and, to a lesser degree, on functionality. Performance testing is carried out through simulation and evaluation, whereas functional testing considers some program characteristics (such as specification and structure). Despite the type of testing, the majority of efforts are focused at the unit and integration test levels of the specific MapReduce functions without considering other parts of the technology stack. Conclusions Researchers have both opportunities and challenges in performance and functional testing, and there is room to improve their research though the use of mature and standard validation methods.

show abstract

Section: Resultsmentioning

confidence: 99%

Testing MapReduce programs: A systematic mapping study

Morán

Riva

Tuya

2018

J Software Evolu Process

View full text Add to dashboard Cite

show abstract

“…d n £ can be generated by composing the two aforementioned permutations corresponding to S 1 and S 2 respectively (cf. Proposition 1 in [3]). The construction of S 1 (resp.…”

Section: Streaming Numerical Transducersmentioning

confidence: 97%

“…Note that we do not allow multiplications in the language, so the reduction from the Diophantine equations in [3] no longer works. Even though, if we do not distinguish the control and data variables, we can show easily that commutativity problem for this language is still undecidable, by a reduction from the reachability problem of Petri nets with inhibitor arcs [12,15].…”

Section: Language For Integer Reducersmentioning

confidence: 99%

The Commutativity Problem of the MapReduce Framework: A Transducer-Based Approach

Chen

Song

Zhang

2016

Computer Aided Verification

Self Cite

View full text Add to dashboard Cite

Abstract. MapReduce is a popular programming model for data parallel computation. In MapReduce, the reducer produces an output from a list of inputs. Due to the scheduling policy of the platform, the inputs may arrive at the reducers in different order. The commutativity problem of reducers asks if the output of a reducer is independent of the order of its inputs. Although the problem is undecidable in general, the MapReduce programs in practice are usually used for data analytics and thus require very simple control flow. By exploiting the simplicity, we propose a programming language for reducers where the commutativity problem is decidable. The main idea of the reducer language is to separate the control and data flow of programs and disallow arithmetic operations in the control flow. The decision procedure for the commutativity problem is obtained through a reduction to the equivalence problem of streaming numerical transducers (SNTs), a novel automata model over infinite alphabets introduced in this paper. The design of SNTs is inspired by streaming transducers (Alur and Cerny, POPL 2011). Nevertheless, the two models are intrinsically different since the outputs of SNTs are integers while those of streaming transducers are data words. The decidability of the equivalence of SNTs is achieved with an involved combinatorial analysis of the evolvement of the values of the integer variables during the runs of SNTs.

show abstract

“…A study of 507 MapReduce programs in production reveals at least 5 different kinds of these faults [30]. To detect them, Csallner et al [31] and Chen et al [32] use testing techniques based on symbolic execution and model checking. Other authors [21], [33] identified and classified more kinds of these faults that are caused by the infrastructure configurations.…”

Section: Related Workmentioning

confidence: 99%

Towards Ex Vivo Testing of MapReduce Applications

Morán

Bertolino

Riva

et al. 2017

2017 IEEE International Conference on Software Quality, Reliability and Security (QRS)

View full text Add to dashboard Cite

Abstract-Big Data programs are those that process large data exceeding the capabilities of traditional technologies. Among newly proposed processing models, MapReduce stands out as it allows the analysis of schema-less data in large distributed environments with frequent infrastructure failures. Functional faults in MapReduce are hard to detect in a testing/preproduction environment due to its distributed characteristics. We propose an automatic test framework implementing a novel testing approach called Ex Vivo. The framework employs data from production but executes the tests in a laboratory to avoid side-effects on the application. Faults are detected automatically without human intervention by checking if the same data would generate different outputs with different infrastructure configurations. The framework (MrExist) is validated with a real-world program. MrExist can identify a fault in a few seconds, then the program can be stopped, not only avoiding an incorrect output, but also saving money, time and energy of production resources.

show abstract

Commutativity of Reducers

Cited by 18 publications

References 18 publications

Testing MapReduce programs: A systematic mapping study

Testing MapReduce programs: A systematic mapping study

The Commutativity Problem of the MapReduce Framework: A Transducer-Based Approach

Towards Ex Vivo Testing of MapReduce Applications

Contact Info

Product

Resources

About