Counterexample Interpretation for Contract-Based Design

Kaleeswaran, Arut Prakash; Nordmann, Arne; Vogel, Thomas; Grunske, Lars

doi:10.1007/978-3-030-58920-2_7

Cited by 6 publications

(10 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To do this, it compares the generated counterexamples with the correct behaviors of the verification model, and differentiates the error transitions and correct transitions in the counterexample. Similarly, the approach by Kaleeswaran et al [96] identifies erroneous and correct states in the counterexample. It particularly supports engineers in locating faults in a Contract-Based Design (CBD), a user-provided design model, that does not pass the refinement check.…”

Section: Minimized Counterexamplementioning

confidence: 99%

A systematic literature review on counterexample explanation

Kaleeswaran,

Nordmann,

Vogel

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Context: Safety is of paramount importance for cyber-physical systems in domains such as automotive, robotics, and avionics. Formal methods such as model checking are one way to ensure the safety of cyber-physical systems. However, adoption of formal methods in industry is hindered by usability issues, particularly the difficulty of understanding model checking results. Objective: We want to provide an overview of the state of the art for counterexample explanation by investigating the contexts, techniques, and evaluation of research approaches in this field. This overview shall provide an understanding of current and guide future research. Method: To provide this overview, we conducted a systematic literature review. The survey comprises 116 publications that address counterexample explanations for model checking. Results: Most primary studies provide counterexample explanations graphically or as traces, minimize counterexamples to reduce complexity, localize errors in the models expressed in the input formats of model checkers, support linear temporal logic or computation tree logic specifications, and use model checkers of the Smybolic Model Verifier family. Several studies evaluate their approaches in safety-critical domains with industrial applications. Conclusion: We notably see a lack of research on counterexample explanation that targets probabilistic and real-time systems, leverages the explanations to domain-specific models, and evaluates approaches in user studies. We conclude by discussing the adequacy of different types of explanations for users with varying domain and formal methods expertise, showing the need to support laypersons in understanding model checking results to increase adoption of formal methods in industry.

show abstract

Section: Minimized Counterexamplementioning

confidence: 99%

A systematic literature review on counterexample explanation

Kaleeswaran,

Nordmann,

Vogel

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Our counterexample explanation approach focuses on enhancing safety analysis for automotive systems [3]. Thus, we are interested in performing this user study only with Bosch automotive engineers, particularly engineers working on system development, requirement elicitation, and safety analysis.…”

Section: Participantsmentioning

confidence: 99%

“…1 and Sect. 2 in [3] for an example). Whenever an inconsistency is found during the verification, the model checker exemplifies the violation by generating a counterexample.…”

Section: Introductionmentioning

confidence: 99%

“…They are used to specify requirements as pattern-based specifications to support verification during product development [10], [11]. Additionally, we have presented a counterexample explanation approach that attempts to ease the use of formal methods by reducing the manual work and difficulty of interpreting the verification results generated by model checkers [3]. Particularly, we target refinements in system design and the verification of their consistency.…”

Section: Introductionmentioning

confidence: 99%

“…To explore whether our counterexample explanation approach [3] does improve the understanding of the model checking results, we intend to perform a one-group pretestposttest experiment. Since we want to perform the study with professional engineers working at Bosch, our study requires working time from these engineers and thus implicitly incurs costs.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A User-Study Protocol for Evaluation of Formal Verification Results and their Explanation

Kaleeswaran¹,

Nordmann²,

Vogel³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

A user study for evaluation of formal verification results and their explanation at Bosch

Kaleeswaran,

Nordmann,

Vogel

et al. 2023

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

Context Ensuring safety for any sophisticated system is getting more complex due to the rising number of features and functionalities. This calls for formal methods to entrust confidence in such systems. Nevertheless, using formal methods in industry is demanding because of their lack of usability and the difficulty of understanding verification results. Objective We evaluate the acceptance of formal methods by Bosch automotive engineers, particularly whether the difficulty of understanding verification results can be reduced. Method We perform two different exploratory studies. First, we conduct a user survey to explore challenges in identifying inconsistent specifications and using formal methods by Bosch automotive engineers. Second, we perform a one-group pretest-posttest experiment to collect impressions from Bosch engineers familiar with formal methods to evaluate whether understanding verification results is simplified by our counterexample explanation approach. Results The results from the user survey indicate that identifying refinement inconsistencies, understanding formal notations, and interpreting verification results are challenging. Nevertheless, engineers are still interested in using formal methods in real-world development processes because it could reduce the manual effort for verification. Additionally, they also believe formal methods could make the system safer. Furthermore, the one-group pretest-posttest experiment results indicate that engineers are more comfortable understanding the counterexample explanation than the raw model checker output. Limitations The main limitation of this study is the generalizability beyond the target group of Bosch automotive engineers.

show abstract

Counterexample Interpretation for Contract-Based Design

Cited by 6 publications

References 26 publications

A systematic literature review on counterexample explanation

A systematic literature review on counterexample explanation

A User-Study Protocol for Evaluation of Formal Verification Results and their Explanation

A user study for evaluation of formal verification results and their explanation at Bosch

Contact Info

Product

Resources

About