Testing and validating machine learning classifiers by metamorphic testing

Abstract

Machine Learning algorithms have provided core functionality to many application domains - such as bioinformatics, computational linguistics, etc. However, it is difficult to detect faults in such applications because often there is no “test oracle” to verify the correctness of the computed outputs. To help address the software quality, in this paper we present a technique for testing the implementations of machine learning classification algorithms which support such applications. Our approach is based on the technique “metamorphic testing”, which has been shown to be effective to alleviate the oracle problem. Also presented include a case study on a real-world machine learning application framework, and a discussion of how programmers implementing machine learning algorithms can avoid the common pitfalls discovered in our study. We also conduct mutation analysis and cross-validation, which reveal that our method has high effectiveness in killing mutants, and that observing expected cross-validation result alone is not sufficiently effective to detect faults in a supervised classification program. The effectiveness of metamorphic testing is further confirmed by the detection of real faults in a popular open-source classification program.

A Survey on Metamorphic Testing

“…For most problems, a variety of metamorphic relations with different fault-detection capability can be identified [9], [16], [18], [27], [28], [29], [30], [31], [32], [33], [34], [35]. Therefore, it is advisable to use a variety of diverse metamorphic relations to effectively test a given program.…”

Towards Ex Vivo Testing of MapReduce Applications

“…This kind of oracle can detect some faults without any knowledge about the expected output. The oracle employed in MrExist is automatically derived from the program executions [71] using metamorphic testing [72]- [74], that is a field also employed to test machine learning programs [75] and in In Vivo frameworks [76] In most metamorphic testing research, the test cases are generated with random testing [77]. In MrExist, the original test cases are also obtained randomly based on a sampling of the production dataset.…”

How to test bioinformatics software?

“…If a MR is violated, for any pair of source and follow-up test cases, the tester reports a failure in the program. MT has been successfully applied to test many different types of software, such as numerical programs (Zhou et al 2004), embedded software (Kuo et al 2011), analysis of feature models (Segura et al 2010), machine learning Xie et al 2011), testing service oriented applications (Chan et al 2007), and big data analytics (Otero and Peter 2015). A simple and classical example of MT is to test the correctness of an implementation of a program that computes the sin(x) trigonometric function, using some wellknown mathematical properties of the function as MRs (Table 3).…”