Testing vision-based control systems using learnable evolutionary algorithms

Abstract

Vision-based control systems are key enablers of many autonomous vehicular systems, including self-driving cars. Testing such systems is complicated by complex and multidimensional input spaces. We propose an automated testing algorithm that builds on learnable evolutionary algorithms. These algorithms rely on machine learning or a combination of machine learning and Darwinian genetic operators to guide the generation of new solutions (test scenarios in our context). Our approach combines multiobjective populationbased search algorithms and decision tree classification models to achieve the following goals: First, classification models guide the search-based generation of tests faster towards critical test scenarios (i.e., test scenarios leading to failures). Second, search algorithms refine classification models so that the models can accurately characterize critical regions (i.e., the regions of a test input space that are likely to contain most critical test scenarios). Our evaluation performed on an industrial automotive vision-based control system shows that: (1) Our algorithm outperforms a baseline evolutionary search algorithm and generates 78% more distinct, critical test scenarios compared to the baseline algorithm. (2) Our algorithm accurately characterizes critical regions of the system under test, thus identifying the conditions that likely to lead to system failures.

Deploying Search Based Software Engineering with Sapienz at Facebook

“…This impact is being felt, not only at Facebook, but elsewhere, such as Ericsson [2], Google [77], and Microsoft [70] as well as the earlier pioneering work of Wegener and his colleagues at Daimler [72]. SBSE has been applied to test embedded systems software in the Automotive domain, for example with industrial case studies involving the Ford Motor Company, Delphi Technologies and IEE S.A. [1,54,61], and the space domain at SES S.A. [69]. It has also been deployed in the financial services sector for security testing [42] and in the maritime sector at Kongsberg Gruppen for stress testing [3].…”

Better software analytics via “DUO”: Data mining algorithms using/used-by optimizers

“…For discrete classes the Gini or entropy measures can assess diversity. Decision tree learners are widely applied in software engineering due to the simplicity and interpretability [1,4,15,21,23,35,58,68,73,74,76,90,95,98]. Support Vector Machines (SVMs) are supervised learning trying to separate training items from two classes by a clear gap [93].…”

Testing autonomous cars for feature interaction failures using many-objective search

“…Testing autonomous cars. Search-based approaches have been used for black-box testing of driver-assistance features [13,14,22,23]. Bühler and Wegener use a single-objective search algorithm to test a vehicle-to-vehicle braking assistance [23] and an autonomous parking feature [22].…”

“…al. rely on multi-objective search [13] and learnable evolutionary algorithms [14] to generate test cases violating safety requirements of self-driving systems. Recently, Tian et.…”

“…Specifically, we aim to generate test inputs that expose undesired feature interactions when applied to executable function models of self-driving systems. Search-based techniques have been successfully applied to simulation-based testing of control systems and self-driving features [3,13,14,23,53,54] as well as various other testing problems such as unit testing [38,55,69], regression testing [49,73] and optimizing machine learning components [66].…”