ARCH-COMP 2019 Category Report: Falsification

Abstract

This report presents the results from the 2019 friendly competition in the ARCH workshop for the falsification of temporal logic specifications over Cyber-Physical Systems. We describe the organization of the competition and how it differs from previous years. We give background on the participating teams and tools and discuss the selected benchmarks and results. The benchmarks are available on the ARCH website1, as well as in the competition’s gitlab repository2. The main outcome of the 2019 competition is a common benchmark repository, and an initial base-line for falsification, with results from multiple tools, which will facilitate comparisons and tracking of the state-of-the-art in falsification in the future.

Falsification of cyber-physical systems with robustness-guided black-box checking

“…We also show the result of a pure random sampling process (PureRandom) to confirm the hardness of the benchmarks. We also note that φ 1 and φ 7 contain AT1 and a variant of AT5 specifications in [20]. Both of the specifications are falsified by GA 10 times out of 10 trials.…”

“…It is also a future work to conduct further detailed experimental evaluation to compare with more tools by using the ARCH-COMP benchmark [20], or to optimize some parameters. For example, for the alphabet size, there should exist a trade-off between the computation cost and covering a larger class of signals.…”

“…These benchmarks are mainly used to compare the scalability with respect to the size of the benchmark. (5,20), (5,25), (15,30), 26] (v < p 1 ) ∨ □ [28,28] (v > p 2 )…”

Multi-armed Bandits for Boolean Connectives in Hybrid System Falsification

“…The specifications concern the relation between the three output signals to check whether the car is subject to some unexpected or unsafe behaviors. The second benchmark is the Abstract Fuel Control (AFC) model [16,25]. It takes two input signals, pedal angle∈ [8.8, 90] and engine speed∈[900, 1100], and outputs the critical signal airfuel ratio (AF ), which influences fuel efficiency and car performance.…”

“…The specifications check whether this property holds under both normal mode and power enrichment mode. The third benchmark is a model of a magnetic levitation system with a NARMA-L2 neurocontroller (NN) [7,16]. It takes one input signal, Ref ∈ [1,3], which is the reference for the output signal Pos, the position of a magnet suspended above an electromagnet.…”