A User-Study Protocol for Evaluation of Formal Verification Results and their Explanation

Kaleeswaran, Arut Prakash; Nordmann, Arne; Vogel, Thomas; Grunske, Lars

doi:10.48550/arxiv.2108.06376

Cited by 1 publication

(1 citation statement)

References 15 publications

(27 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We pre-registered our studies (Part 1 and 2) by a report submitted to and accepted by the Registered Reports Track at the International Conference on Software Maintenance and Evolution (ICSME) 2021 1 (Kaleeswaran et al 2021). Both studies have been conducted following the same research protocol as described in the registered report (Kaleeswaran et al 2021) without any modification to the design of the studies.…”

Section: Contributionsmentioning

confidence: 99%

A user study for evaluation of formal verification results and their explanation at Bosch

Kaleeswaran,

Nordmann,

Vogel

et al. 2023

Empir Software Eng

View full text Add to dashboard Cite

Context Ensuring safety for any sophisticated system is getting more complex due to the rising number of features and functionalities. This calls for formal methods to entrust confidence in such systems. Nevertheless, using formal methods in industry is demanding because of their lack of usability and the difficulty of understanding verification results. Objective We evaluate the acceptance of formal methods by Bosch automotive engineers, particularly whether the difficulty of understanding verification results can be reduced. Method We perform two different exploratory studies. First, we conduct a user survey to explore challenges in identifying inconsistent specifications and using formal methods by Bosch automotive engineers. Second, we perform a one-group pretest-posttest experiment to collect impressions from Bosch engineers familiar with formal methods to evaluate whether understanding verification results is simplified by our counterexample explanation approach. Results The results from the user survey indicate that identifying refinement inconsistencies, understanding formal notations, and interpreting verification results are challenging. Nevertheless, engineers are still interested in using formal methods in real-world development processes because it could reduce the manual effort for verification. Additionally, they also believe formal methods could make the system safer. Furthermore, the one-group pretest-posttest experiment results indicate that engineers are more comfortable understanding the counterexample explanation than the raw model checker output. Limitations The main limitation of this study is the generalizability beyond the target group of Bosch automotive engineers.

show abstract

Section: Contributionsmentioning

confidence: 99%