Using Machine Learning to Build Test Oracles: an Industrial Case Study on Elevators Dispatching Algorithms

Abstract

The software of elevators requires maintenance over several years to deal with new functionality, correction of bugs or legislation changes. To automatically validate this software, test oracles are necessary. A typical approach in industry is to use regression oracles. These oracles have to execute the test input both, in the software version under test and in a previous software version. This practice has several issues when using simulation to test elevators dispatching algorithms at system level. These issues include a long test execution time and the impossibility of re-using test oracles both at different test levels and in operation. To deal with these issues, we propose DARIO, a test oracle that relies on regression learning algorithms to predict the Qualify of Service of the system. The regression learning algorithms of this oracle are trained by using data from previously tested versions. An empirical evaluation with an industrial case study demonstrates the feasibility of using our approach in practice. A total of five regression learning algorithms were validated, showing that the regression tree algorithm performed best. For the regression tree algorithm, the accuracy when predicting verdicts by DARIO ranged between 79 to 87%.

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

“…Recently, however, [84,86,88,95] generate oracles for functions with unconstrainede.g., integer-output.…”

“…[84, 86] compared regression trees, SVM, an ensemble model, a Regression Gaussian Process (RGP), and a stepwise regression. [84] found regression tree to be the best, while [86] found regression tree, ensemble, and RGP valid.…”

“…Ensembles are often able to reach stable, accurate conclusions in situations where a single model may be inaccurate. A small number of studies have applied ensembles [27,63,75,84,86], but such techniques are rare. Ensembles may be a way to overcome the fragility of some ML approaches.…”

Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions

“…A variety of manually defined fitness functions was proposed in the literature. Manually defined fitness functions can guide the search to produce test outputs with diverse shapes [63], maximize or minimize the outputs of a system characterizing its critical behavior [20], maximize diversity in output signals [64], quantify the difference between a reference and an output signal [15], and minimize the difference between the expected and simulated behaviour of a CPS [46]. Many manually defined fitness functions were also proposed to guide SBST frameworks that analyze software code.…”

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

“…4 Unfortunately, CPSs are not exempt of non-functional performance bugs, which can occasionally lead to more severe consequences, such as functional failures. In our conference-version paper published on Automation of Software Test (AST) conference, 5 we explored the use of machine learning (ML) to tackle the test oracle problem. Specifically, we proposed Dispatching AlgoRIthm Oracle (DARIO), a test oracle that relies on regression learning algorithms to automatically validate elevators dispatching algorithms.…”