Generating Complex and Faulty Test Data through Model-Based Mutation Analysis

Abstract

Testing the correct behaviour of data processing systems in the presence of faulty data is extremely expensive. The data structures processed by these systems are often complex, with many data fields and multiple constraints among them. Software engineers, in charge of testing these systems, have to handcraft complex data files or databases, while ensuring compliance with the multiple constraints to prevent the generation of trivially invalid inputs. In addition, assessing test results often means analysing complex output and log data. Though many techniques have been proposed to automatically test systems based on models, little exists in the literature to support the testing of systems where the complexity is in the data consumed in input or produced in output, with complex constraints between them. In particular, such systems often need to be tested with the presence of faults in the input data, in order to assess the robustness and behaviour of the system in response to such faults. This paper presents an automated test technique that relies upon six generic mutation operators to automatically generate faulty data. The technique receives two inputs: field data and a data model, i.e. a UML class diagram annotated with stereotypes and OCL constraints. The annotated class diagram is used to tailor the behaviour of the generic mutation operators to the fault model that is assumed for the system under test and the environment in which it is deployed.Empirical results obtained with a large data acquisition system in the satellite domain show that our approach can successfully automate the generation of test suites that achieve slightly better instruction coverage than manual testing based on domain expertise.Further, the technique automates test oracles that, in our 978-1-4799-7125-1/15/$31.00 ©2015 IEEE

Augmenting Field Data for Testing Systems Subject to Incremental Requirements Changes

“…In the case of SES-DAQ, the process is automated by using a parser that follows the approach described in [Di Nardo et al 2015a]. The parser uses stereotypes in the UML class diagram to determine the mapping between class attributes and the bytes stored on disk.…”

“…To define data requirements, we rely upon a data modelling methodology described in [Di Nardo et al 2013;Di Nardo et al 2015a] that uses UML class diagrams to capture the structure of inputs and outputs, relies upon Object Constraint Language (OCL) [OMG 2015] expressions to define relationships between the inputs and outputs, and uses UML stereotypes and OCL expressions to capture a fault model.…”

“…To limit the modelling effort, other approaches generate test inputs by mutating existing field data [Shan and Zhu 2009;Di Nardo et al 2015a]. The technique described in [Di Nardo et al 2015a], for example, generates test inputs by mutating available field data represented by a data model.…”

“…Additionally, the newly generated test inputs can be further processed by other automated testing techniques. For example, the mutation technique that we introduced in [Di Nardo et al 2015a] can be used to automatically modify the synthetic field data generated to perform robustness testing. Also, the outputs that result from the execution of the software under test against the synthesised field data can be automatically evaluated by relying upon a technique that makes use of data constraints in the data model [Di Nardo et al 2013].…”

“…The technique described in [Di Nardo et al 2015a], for example, generates test inputs by mutating available field data represented by a data model. This technique does not require exhaustive modelling of all the characteristics of the test inputs; however, it is not applicable when existing field data do not comply with new data requirements.…”