Sound black-box checking in the LearnLib

Meijer, Jeroen; Pol, Jaco van de

doi:10.1007/s11334-019-00342-6

Cited by 8 publications

(8 citation statements)

References 36 publications

(47 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…LBT does not rely on PAC-learning and does not provide provable probabilistic guarantees on the hypothesis. Somehow, this issue has been partially studied in [49] but at the price of relaxing the black-box setting by observing and storing the SUT internal state.…”

Section: Related Workmentioning

confidence: 99%

Property Checking with Interpretable Error Characterization for Recurrent Neural Networks

Mayr

Yovine

Visca

2021

MAKE

View full text Add to dashboard Cite

This paper presents a novel on-the-fly, black-box, property-checking through learning approach as a means for verifying requirements of recurrent neural networks (RNN) in the context of sequence classification. Our technique steps on a tool for learning probably approximately correct (PAC) deterministic finite automata (DFA). The sequence classifier inside the black-box consists of a Boolean combination of several components, including the RNN under analysis together with requirements to be checked, possibly modeled as RNN themselves. On one hand, if the output of the algorithm is an empty DFA, there is a proven upper bound (as a function of the algorithm parameters) on the probability of the language of the black-box to be nonempty. This implies the property probably holds on the RNN with probabilistic guarantees. On the other, if the DFA is nonempty, it is certain that the language of the black-box is nonempty. This entails the RNN does not satisfy the requirement for sure. In this case, the output automaton serves as an explicit and interpretable characterization of the error. Our approach does not rely on a specific property specification formalism and is capable of handling nonregular languages as well. Besides, it neither explicitly builds individual representations of any of the components of the black-box nor resorts to any external decision procedure for verification. This paper also improves previous theoretical results regarding the probabilistic guarantees of the underlying learning algorithm.

show abstract

Section: Related Workmentioning

confidence: 99%

Property Checking with Interpretable Error Characterization for Recurrent Neural Networks

Mayr

Yovine

Visca

2021

MAKE

View full text Add to dashboard Cite

show abstract

“…BBC has been used for testing numerical software [28], distributed systems [29], and autonomous systems [23]. BBC is implemented in LBTest [30] and LearnLib [22,27].…”

Section: Related Workmentioning

confidence: 99%

Efficient Black-Box Checking via Model Checking with Strengthened Specifications

Shijubo¹,

Waga²,

Suenaga³

2021

Preprint

View full text Add to dashboard Cite

Black-box checking (BBC) is a testing method for cyberphysical systems (CPSs) as well as software systems. BBC consists of active automata learning and model checking; a Mealy machine is learned from the system under test (SUT), and the learned Mealy machine is verified against a specification using model checking. When the Mealy machine violates the specification, the model checker returns an input witnessing the specification violation of the Mealy machine. We use it to refine the Mealy machine or conclude that the SUT violates the specification. Otherwise, we conduct equivalence testing to find an input witnessing the difference between the Mealy machine and the SUT. In the BBC for CPSs, equivalence testing tends to be time-consuming due to the time for the system execution. In this paper, we enhance the BBC utilizing model checking with strengthened specifications. By model checking with a strengthened specification, we have more chance to obtain an input witnessing the specification violation than model checking with the original specification. The refinement of the Mealy machine with such an input tends to reduce the number of equivalence testing, which improves the efficiency. We conducted experiments with an automotive benchmark. Our experiment results demonstrate the merit of our method.

show abstract

“…The correctness of the VC algorithm relies on the upper bound of the size of the state space of the black-box system. In [38], Büchi acceptance condition in the state space of the black-box system is used for the sound equivalence checking.…”

Section: Related Workmentioning

confidence: 99%

“…However, up to our knowledge, there is no work exploiting the quantitative satisfaction degree of the requirements in addition to Boolean satisfaction. For BBC, as far as we are aware of, two tools have been presented: LBTest [41] and an implementation [38] in LearnLib [28]. Our prototypical tool FalCAuN relies on the implementation [38] in LearnLib.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Falsification of cyber-physical systems with robustness-guided black-box checking

Waga

2020

Proceedings of the 23rd International Conference on Hybrid Systems: Computation and Control

View full text Add to dashboard Cite

For exhaustive formal verification, industrial-scale cyberphysical systems (CPSs) are often too large and complex, and lightweight alternatives (e.g., monitoring and testing) have attracted the attention of both industrial practitioners and academic researchers. Falsification is one popular testing method of CPSs utilizing stochastic optimization. In stateof-the-art falsification methods, the result of the previous falsification trials is discarded, and we always try to falsify without any prior knowledge. To concisely memorize such prior information on the CPS model and exploit it, we employ Black-box checking (BBC), which is a combination of automata learning and model checking. Moreover, we enhance BBC using the robust semantics of STL formulas, which is the essential gadget in falsification. Our experiment results suggest that our robustness-guided BBC outperforms a state-of-the-art falsification tool. CCS CONCEPTS• Computer systems organization → Embedded and cyber-physical systems; Real-time system specification; • Software and its engineering → Formal software verification; Search-based software engineering.

show abstract

Sound black-box checking in the LearnLib

Cited by 8 publications

References 36 publications

Property Checking with Interpretable Error Characterization for Recurrent Neural Networks

Property Checking with Interpretable Error Characterization for Recurrent Neural Networks

Efficient Black-Box Checking via Model Checking with Strengthened Specifications

Falsification of cyber-physical systems with robustness-guided black-box checking

Contact Info

Product

Resources

About