I8K|DQ-BigData: I8K Architecture Extension for Data Quality in Big Data

Rivas, Bibiano; Merino, Jorge; Serrano, Manuel A.; Caballero, Ismael; Piattini, Mario

doi:10.1007/978-3-319-25747-1_17

Cited by 6 publications

(5 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to evaluate the proposed approach, we use as case study the MapReduce program described in I8K|DQ-BigData framework [13]. This program measures the quality of the data exchanged between organizations according to part 140 of the ISO/TS 8000 [14].…”

Section: Case Studymentioning

confidence: 99%

Infrastructure-Aware Functional Testing of MapReduce Programs

Morán

Rivas

Riva

et al. 2016

2016 IEEE 4th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW)

Self Cite

View full text Add to dashboard Cite

Abstract-Programs that process a large volume of data generally run in a distributed and parallel architecture, such as the programs implemented in the processing model MapReduce. In these programs, developers can abstract the infrastructure where the program will run and focus on the functional issues. However, the infrastructure configuration and its state cause different parallel executions of the program and some could derive in functional faults which are hard to reveal. In general, the infrastructure that executes the program is not considered during the testing, because the tests usually contain few input data and then the parallelization is not necessary. In this paper a testing technique is proposed to generate different infrastructure configurations for a given test input data, and then the program is executed in these configurations in order to reveal functional faults. This testing technique is automatized by using a test engine and applied in a case study. As a result, several infrastructure configurations are automatically generated and executed for a test case revealing a functional fault that is then fixed by the developer.

show abstract

Section: Case Studymentioning

confidence: 99%

Infrastructure-Aware Functional Testing of MapReduce Programs

Morán

Rivas

Riva

et al. 2016

2016 IEEE 4th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The program fails when data about the same both user and item are unsorted due the parallel execution of the framework. (P2) Data quality analysis [44] measures the quality of the data interchanged by companies. The program fails when some particular data under analysis are not processed in the same order as in the input due to parallelization issues done by the framework.…”

Section: Discussionmentioning

confidence: 99%

Automatic Debugging of Design Faults in MapReduce Applications

Morán,

Bertolino,

de la Riva

et al. 2024

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Among the current technologies to analyse large data, the MapReduce processing model stands out in Big Data. MapReduce is implemented in frameworks such as Hadoop, Spark or Flink that are able to manage the program executions according to the resources available at runtime. The developer should design the program in order to support all possible nondeterministic executions. However, the program may fail due to a design fault. Debugging these kinds of faults is difficult because the data are executed non-deterministically in parallel and the fault is not caused directly by the code, but by its design. This paper presents a framework called MRDebug which includes two debugging techniques focused on the MapReduce design faults. A spectrum-based fault localization technique locates the root cause of these faults analysing several executions of the test case, and a Delta Debugging technique isolates the data relevant to trigger the failure. An empirical evaluation with 13 programs shows that MRDebug is effective in debugging the faults, especially when the localization is done with the reduced data. In summary, MRDebug automatically provides valuable information to understand MapReduce design faults as it helps locate their root cause and obtains a minimal data that triggers the failure.

show abstract

“…This program could fail when the data of each user-item is split and parallelized. 2) Data quality analysis [80]: Measure of the quality of data interchanged between companies, based on international standards. This program did not correctly track the measurements and they could be incorrectly assigned due the parallel execution.…”

Section: ) Effectiveness: Setupmentioning

confidence: 99%

Automatic Testing of Design Faults in MapReduce Applications

et al. 2018

View full text Add to dashboard Cite

New processing models are being adopted in Big Data Engineering to overcome the limitations of traditional technology. Among them, MapReduce stands out by allowing for the processing of large volumes of data over a distributed infrastructure that can change during runtime. The developer only designs the functionality of the program and its execution is managed by a distributed system. As a consequence, a program can behave differently at each execution because it is automatically adapted to the resources available at each moment. Therefore, when the program has a design fault, this could be revealed in some executions and masked in others. However, during testing, these faults are usually masked because the test infrastructure is stable, and they are only revealed in production because the environment is more aggressive with infrastructure failures, among other reasons. This paper proposes new testing techniques aimed to detect these design faults by simulating different infrastructure configurations. The testing techniques generate a representative set of infrastructure configurations that as whole are more likely to reveal failures using Random testing, and Partition testing together with Combinatorial testing. The techniques are automated by using a test execution engine called MRTest that is able to detect these faults using only the test input data, regardless of the expected output. Our empirical evaluation shows that MRTest can automatically detect these design faults within a reasonable time.

show abstract

I8K|DQ-BigData: I8K Architecture Extension for Data Quality in Big Data

Cited by 6 publications

References 2 publications

Infrastructure-Aware Functional Testing of MapReduce Programs

Infrastructure-Aware Functional Testing of MapReduce Programs

Automatic Debugging of Design Faults in MapReduce Applications

Automatic Testing of Design Faults in MapReduce Applications

Contact Info

Product

Resources

About