Online Model-Based Testing under Uncertainty

Camilli, Matteo; Bellettini, Carlo; Gargantini, Angelo; Scandurra, Patrizia

doi:10.1109/issre.2018.00015

Cited by 22 publications

(16 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The three selected strategies are the uncertainty-aware test case generation, a random walk test case generation approach and a history-based approach. We let the reader refer to the detailed description provided by Camilli et al [9] for further information about the two latter traditional MBT methods. Data collected during operation are shown in Figure 8 (i.e., coverage and accuracy) and Figure 9 (i.e., effort).…”

Section: Results For Rq2mentioning

confidence: 99%

“…As introduced by Camilli et al [9], the execution of a generic program P can be viewed in the same way as the one of its formal specification. In particular, the MDP model extracted from the SUT execution is called model program and denoted by M SUT .…”

Section: Preliminary Definitionsmentioning

confidence: 99%

“…Our major objective is to show the effectiveness of METRIC in statistical hypothesis testing (rather than functional testing) of uncertain software systems by measuring both the accuracy and the effort of the inference process. We show a comparative evaluation between our approach and traditional pseudorandom model-based test case generation algorithms, thus showing the convenience of METRIC.This approach has been introduced by Camilli et al [5,9,10] through a preliminary sketch of a testing method under uncertainty supported by a prototypal software implementation. Here, we provide an extended presentation of the approach as part of a comprehensive methodology.…”

mentioning

confidence: 99%

“…This approach has been introduced by Camilli et al [5,9,10] through a preliminary sketch of a testing method under uncertainty supported by a prototypal software implementation. Here, we provide an extended presentation of the approach as part of a comprehensive methodology.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Model‐based hypothesis testing of uncertain software systems

Camilli

Gargantini

Scandurra

2020

Software Testing Verif & Rel

Self Cite

View full text Add to dashboard Cite

Nowadays, there exists an increasing demand for reliable software systems able to fulfill their requirements in different operational environments and to cope with uncertainty that can be introduced both at design-time and at runtime because of the lack of control over third-party system components and complex interactions among software, hardware infrastructures and physical phenomena. This article addresses the problem of the discrepancy between measured data at runtime and the design-time formal specification by using an inverse uncertainty quantification approach. Namely, we introduce a methodology called METRIC and its supporting toolchain to quantify and mitigate software system uncertainty during testing by combining (on-the-fly) model-based testing and Bayesian inference. Our approach connects probabilistic input/output conformance theory with statistical hypothesis testing in order to assess if the behaviour of the system under test corresponds to its probabilistic formal specification provided in terms of a Markov decision process. An uncertainty-aware model-based test case generation strategy is used as a means to collect evidence from software components affected by sources of uncertainty. Test results serve as input to a Bayesian inference process that updates beliefs on model parameters encoding uncertain quality attributes of the system under test. This article describes our approach from both theoretical and practical perspectives. An extensive empirical evaluation activity has been conducted in order to assess the cost-effectiveness of our approach. We show that, under same effort constraints, our uncertainty-aware testing strategy increases the accuracy of the uncertainty quantification process up to 50 times with respect to traditional model-based testing methods. of uncertainty away because of insufficient understanding and unavailability of adequate methods and techniques. Today, endowing conventional software engineering methodologies with techniques and practices able to model, quantify, and manage uncertainty explicitly is becoming increasingly crucial [1,2]. In particular, techniques that explicitly consider uncertainty in software testing is an emerging research direction.This article focuses on the problem of uncertainty quantification and introduces a methodology able to deal with it by means of a model-based hypothesis testing approach. Namely, we use statistical hypothesis testing to reduce the discrepancy between delivered products and initial design-time uncertainty assumptions. As described by Broy at al.[3], model-based testing (MBT) is a software testing technique where runtime behaviour of a software system under test (SUT) is checked against predictions made by a model description of the system's behaviour. In our approach, the design-time model description contains uncertain aspects explicitly specified by means of probability. Uncertainty must be mitigated accounting for evidence during the actual system's execution. Thus, we make use of (on-the-fly) MBT as a means to extrac...

show abstract

Section: Results For Rq2mentioning

confidence: 99%

Section: Preliminary Definitionsmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Model‐based hypothesis testing of uncertain software systems

Camilli

Gargantini

Scandurra

2020

Software Testing Verif & Rel

Self Cite

View full text Add to dashboard Cite

show abstract

“…Несколько статей относятся к встроенным тестам для компонентных систем с акцентом на их архитектуру. В работе [26] использовался онлайн-подход к тестированию на базе моделей, где тестируемая система (SUT) стимулировалась марковским процессом принятия решений (MDP). Состояния модели и поведение системы были связаны, чтобы генерировать действия модели использования, которые приведут к намеченному поведению.…”

Section: экспериментированиеunclassified

Online Testing of Dynamic Reconfigurations w.r.t. Adaptation Policies

Dadeau

Gros

Kouchnarenko

2021

Model. anal. inf. sist.

View full text Add to dashboard Cite

Self-adaptation of complex systems is a very active domain of research with numerous application domains. Component systems are designed as sets of components that may reconfigure themselves according to adaptation policies, which describe needs for reconfiguration. In this context, an adaptation policy is designed as a set of rules that indicate, for a given set of configurations, which reconfiguration operations can be triggered, with fuzzy values representing their utility. The adaptation policy has to be faithfully implemented by the system, especially w.r.t. the utility occurring in the rules, which are generally specified for optimizing some extra-functional properties (e.g. minimizing resource consumption). In order to validate adaptive systems’ behaviour, this paper presents a model-based testing approach, which aims to generate large test suites in order to measure the occurrences of reconfigurations and compare them to their utility values specified in the adaptation rules. This process is based on a usage model of the system used to stimulate the system and provoke reconfigurations. As the system may reconfigure dynamically, this online test generator observes the system responses and evolution in order to decide the next appropriate test step to perform. As a result, the relative frequencies of the reconfigurations can be measured in order to determine whether the adaptation policy is faithfully implemented. To illustrate the approach the paper reports on experiments on the case study of platoons of autonomous vehicles.

show abstract