Falsification of cyber-physical systems with robustness-guided black-box checking

Abstract

For exhaustive formal verification, industrial-scale cyberphysical systems (CPSs) are often too large and complex, and lightweight alternatives (e.g., monitoring and testing) have attracted the attention of both industrial practitioners and academic researchers. Falsification is one popular testing method of CPSs utilizing stochastic optimization. In stateof-the-art falsification methods, the result of the previous falsification trials is discarded, and we always try to falsify without any prior knowledge. To concisely memorize such prior information on the CPS model and exploit it, we employ Black-box checking (BBC), which is a combination of automata learning and model checking. Moreover, we enhance BBC using the robust semantics of STL formulas, which is the essential gadget in falsification. Our experiment results suggest that our robustness-guided BBC outperforms a state-of-the-art falsification tool. CCS CONCEPTS• Computer systems organization → Embedded and cyber-physical systems; Real-time system specification; • Software and its engineering → Formal software verification; Search-based software engineering.

PAC Model Checking of Black-Box Continuous-Time Dynamical Systems

“…The checking-testing-learning repeated process is costly generally. Recently, a method combining optimization-based falsification and black-box checking was proposed to falsify specifications for black-box cyber-physical systems in [40].…”

ARCH-COMP 2021 Category Report: Falsification with Validation of Results

“…Description. FalCAuN [38] is an experimental tool for testing a Simulink model using blackbox checking [35], an automated testing method based on active automata learning and model checking. In FalCAuN, the input and the output signals of the Simulink model are discretized in time and values, and the model is abstracted into a black-box Mealy machine.…”

“…FalCAuN : Column FR are results obtained when the simulations required for automata learning are not counted, only the simulations required for equivalence testing are capped at 300 (see [38]).…”

“…• We have included some new conjunctive requirements (section 2), which intend to test whether the approaches can deal with multiple-possibly competing-ways in which a particular requirement can be falsified. • There are two new participating tools (section 3), FalCAuN [38] and ForeSee [41]., which means that now we have results of an overall of eight competitors resp. approaches (section 4) • For the first time, we do not only collect statictics on the success rate, but the actual falsifying inputs as produced by the tools, and we validate them independently (section 5).…”

ARCH-COMP 2020 Category Report: Falsification

“…Recent results in falsification have produced a variety of techniques, mature tools, and practical applications, see [3,6] for an overview. Due to the complexity and unclear semantics of Matlab and Simulink models, many previous techniques are entirely black-box and just observe the input/output behavior of the system via simulations, but grey-box approaches have been developed recently [27,1,26] to take some knowledge on the internals of the system into deliberation. This year's falsification competition featured two more tools and participating teams in comparison to the previous year [11] The participating tools 2019 were S-TaLiRo [2], Breach [9], FalStar [28,12], falsify [1], ARIsTEO [23], and zlscheck (based on Zélus [4]), in different configurations (Sec 2).…”