METTLE: A METamorphic Testing Approach to Assessing and Validating Unsupervised Machine Learning Systems

Abstract

Unsupervised machine learning is the training of an artificial intelligence system using information that is neither classified nor labeled, with a view to modeling the underlying structure or distribution in a dataset. Since unsupervised machine learning systems are widely used in many real-world applications, assessing the appropriateness of these systems and validating their implementations with respect to individual users' requirements and specific application scenarios / contexts are indisputably two important tasks. Such assessment and validation tasks, however, are fairly challenging due to the absence of a priori knowledge of the data. In view of this challenge, we develop a METamorphic Testing approach to assessing and validating unsupervised machine LEarning systems, abbreviated as METTLE. Our approach provides a new way to unveil the (possibly latent) characteristics of various machine learning systems, by explicitly considering the specific expectations and requirements of these systems from individual users' perspectives. To support METTLE, we have further formulated 11 generic metamorphic relations (MRs), covering users' generally expected characteristics that should be possessed by machine learning systems. To demonstrate the viability and effectiveness of METTLE, we have performed an experiment involving six commonly used clustering systems. Our experiment has shown that, guided by user-defined MR-based adequacy criteria, end users are able to assess, validate, and select appropriate clustering systems in accordance with their own specific needs. Our investigation has also yielded insightful understanding and interpretation of the behavior of the machine learning systems from an end-user software engineering's perspective, rather than a designer's or implementor's perspective, who normally adopts a theoretical approach.Index Terms-Unsupervised machine learning, clustering assessment, clustering validation, metamorphic testing, metamorphic relation, enduser software engineering.1. In mathematics, the convex hull of a set X of points in the Euclidean plane is the smallest convex set that contains X.

Machine Learning Testing: Survey, Landscapes and Horizons

“…Ramanathan and Pullum [7] proposed a combination of symbolic and statistical approaches to test k-means algorithm, which is a clustering algorithm. Xie et al [119] designed metamorphic relations for unsupervised learning. One paper focused on reinforcement learning testing: Uesato et al [87] proposed a predictive adversarial example generation approach to predict failures and estimate reliable risks in reinforcement learning.…”

Testing machine learning based systems: a systematic mapping

“…Oracle Overall, we found 13 papers in our pool that tackle the oracle problem for MLSs (Zheng et al 2019;Xie et al 2011;Nakajima and Bui 2016, 2019Qin et al 2018;Cheng et al 2018b;Ding et al 2017;Gopinath et al 2018;Murphy et al 2007aSaha and Kanewala 2019;Xie et al 2018). The challenge is to assess the correctness of MLSs' behaviour, which is possibly stochastic, due to the non-deterministic nature of training (e.g., because of the random initialisation of weights or the use of stochastic optimisers) and which depends on the choice of the training set.…”

“…Five works (7%) manipulate only the input data, i.e., they perform input level testing (Bolte et al 2019;Byun et al 2019;Henriksson et al 2019;Wolschke et al 2018). The majority of the papers (64%) operate at the ML model level (model level testing) (Cheng et al 2018a;Ding et al 2017;Du et al 2019;Dwarakanath et al 2018;Eniser et al 2019;Gopinath et al 2018;Groce et al 2014;Guo et al 2018;Kim et al 2019;Li et al 2018;Ma et al 2018bMa et al , c, d, 2019Murphy et al 2007aMurphy et al , b, 2008Murphy et al , b, 2009Nakajima and Bui 2016, 2019Odena et al 2019;Patel et al 2018;Pei et al 2017;Qin et al 2018;Saha and Kanewala 2019;Sekhon and Fleming 2019;Shen et al 2018;Shi et al 2019;Spieker and Gotlieb 2019;Strickland et al 2018;Sun et al 2018a, b;Tian et al 2018;Udeshi and Chattopadhyay 2019;Udeshi et al 2018;Uesato et al 2019;Xie et al 2018Xie et al , 2019Xie et al , 2011Zhang et al 2018aZhang et al , b, 2019Zhao a...…”

“…Nearly half of the analysed papers (56%) propose and evaluate a technique which is domain-agnostic, Fig. 7 Testing Levels i.e., in principle it may be applicable to any MLS (Aniculaesei et al 2018;Byun et al 2019;Cheng et al 2018a, b;Du et al 2019;Eniser et al 2019;Guo et al 2018;Henriksson et al 2019;Kim et al 2019;Li et al 2018;Ma et al 2018bMa et al , c, d, 2019Murphy et al 2007aMurphy et al , b, 2008Murphy et al , b, 2009Nakajima and Bui 2016, 2019Odena et al 2019;Pei et al 2017;Saha and Kanewala 2019;Sekhon and Fleming 2019;Shen et al 2018;Shi et al 2019;Sun et al 2018a, b;Tian et al 2018;Udeshi and Chattopadhyay 2019;Uesato et al 2019;Xie et al 2018Xie et al , 2019Xie et al , 2011Zhang et al 2018aZhang et al , 2019Zhao and Gao 2018). Around 30% proposed approaches are designed for autonomous systems (Abeysirigoonawardena et al 2019;Beglerovic et al 2017;Bühler and Wegener 2004;Klueck et al 2018;Li et al 2016;Mullins et al 2018;de Oliveira Neves et al 2016;Patel et al 2018;Strickland et al 2018;…”

“…In 12 out of 55 (22%) papers the produced artefact is an oracle (Qin et al 2018;Xie et al 2018;Ding et al 2017;Dwarakanath et al 2018, 2009Nakajima and Bui 2016, 2019Saha and Kanewala 2019;Xie et al 2011;Zhang et al 2016). The main focus of 11 papers from this list is a set of metamorphic relationships (MRs), which are then used to generate a metamorphic oracle.…”