On the Danger of Coverage Directed Test Case Generation

Staats, Matt; Gay, Gregory; Whalen, Michael W.; Heimdahl, Mats P. E.

doi:10.1007/978-3-642-28872-2_28

Cited by 55 publications

(48 citation statements)

References 30 publications

(51 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(The Simulink Coder is not a trusted code generator; thus, the generated code must be subjected to rigorous conformance testing demonstrating that no faults were introduced in the code generation. This testing is outside the scope of this paper but in can be fully automated as extensively discussed in our previous work [7,22,25]) Although the process of code generation from the model is straightforward, installing the software on the actual hardware can be a challenge. Below we discuss the challenges we faced while integrating the device with the generated code and discuss the approaches we took to address them.…”

Section: Gpca Implementationmentioning

confidence: 99%

From Requirements to Code: Model Based Development of a Medical Cyber Physical System

Murugesan

Heimdahl

Whalen

et al. 2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The advanced use of technology in medical devices has improved the way health care is delivered to patients. Unfortunately, the increased complexity of modern medical devices poses challenges for development, assurance, and regulatory approval. In an e ort to improve the safety of advanced medical devices, organizations such as FDA have supported exploration of techniques to aid in the development and regulatory approval of such systems. In an ongoing research project, our aim is to provide effective development techniques and exemplars of system development artifacts that demonstrate state of the art development techniques.In this paper we present an end-to-end model-based approach to medical device software development along with the artifacts created in the process. While outlining the approach, we also describe our experiences, challenges, and lessons learned in the process of formulating and analyzing the requirements, modeling the system, formally verifying the models, generating code, and executing the generated code in the hardware for generic patient controlled analgesic infusion pump (GPCA). We believe that the development artifacts and techniques presented in this paper could serve as a generic reference to be used by researchers, practitioners, and authorities while developing and evaluating cyber physical medical devices. Abstract. The advanced use of technology in medical devices has improved the way health care is delivered to patients. Unfortunately, the increased complexity of modern medical devices poses challenges for development, assurance, and regulatory approval. In an effort to improve the safety of advanced medical devices, organizations such as FDA have supported exploration of techniques to aid in the development and regulatory approval of such systems. In an ongoing research project, our aim is to provide effective development techniques and exemplars of system development artifacts that demonstrate state of the art development techniques. In this paper we present an end-to-end model-based approach to medical device software development along with the artifacts created in the process. While outlining the approach, we also describe our experiences, challenges, and lessons learned in the process of formulating and analyzing the requirements, modeling the system, formally verifying the models, generating code, and executing the generated code in the hardware for generic patient controlled analgesic infusion pump (GPCA). We believe that the development artifacts and techniques presented in this paper could serve as a generic reference to be used by researchers, practitioners, and authorities while developing and evaluating cyber physical medical devices.

show abstract

Section: Gpca Implementationmentioning

confidence: 99%

From Requirements to Code: Model Based Development of a Medical Cyber Physical System

Murugesan

Heimdahl

Whalen

et al. 2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…As discussed in the introduction, the effectiveness of coverage criteria is still a very active research topic (Gopinath et al 2014;Inozemtseva and Holmes 2014;Kochhar et al 2015;Staats et al 2012). However, the studies evaluating the effectiveness of coverage measures consider the faults (or mutations) as having same importance and do not take into account their respective probability of failure.…”

Section: Related Workmentioning

confidence: 99%

“…On the other side, we observe that almost all studies assessing the effectiveness of coverage testing have used as a measure of effectiveness the faults (or mutations) detected without distinguishing their probability of failure in use (e.g., Inozemtseva and Holmes 2014;Kochhar et al 2015;Staats et al 2012;Wei et al 2012;Wong et al 1994, just to cite a few). Thus, we still know little about how (and if) coverage testing is related to delivered reliability (Frankl et al 1998).…”

Section: Introductionmentioning

confidence: 99%

“…Coveragebased testing is appealing because it provides in automated way a quantitative feedback from a testing session, i.e., the ratio between the entities covered and their total number. Such coverage measure, used as a supplement to other non-coverage-based testing methods, can be an effective tool (Staats et al 2012), for example to decide whether a test suite derived using another black-box method is adequate, as well as to pinpoint portions of code that have not yet been tested.…”

Section: Introductionmentioning

confidence: 99%

“…A recent large scale experiment (Inozemtseva and Holmes 2014) concludes that high coverage measures achieved by a test suite do not necessarily indicate that the latter also yields high effectiveness. Thus, generating test cases for coverage as a target may be risky, as warned from many sides (e.g., Marick 1999;Staats et al 2012).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing

Miranda

Bertolino²

2017

Software Qual J

View full text Add to dashboard Cite

While the relation between code coverage measures and fault detection is actively studied, only few works have investigated the correlation between measures of coverage and of reliability. In this work, we introduce a novel approach to measuring code coverage, called the operational coverage, that takes into account how much the program's entities are exercised so to reflect the profile of usage into the measure of coverage. Operational coverage is proposed as (i) an adequacy criterion, i.e., to assess the thoroughness of a black box test suite derived from the operational profile, and as (ii) a selection criterion, i.e., to select test cases for operational profile-based testing. Our empirical evaluation showed that operational coverage is better correlated than traditional coverage with the probability that the next test case derived according to the user's profile will not fail. This result suggests that our approach could provide a good stopping rule for operational profile-based testing. With respect to test case selection, our investigations revealed that operational coverage outperformed the traditional one in terms of test suite size and fault detection capability when we look at the average results.

show abstract

Constructing Search Spaces for Search-Based Software Testing Using Neural Networks

Joffe

Clark

2019

Search-Based Software Engineering

View full text Add to dashboard Cite

A central requirement for any Search-Based Software Testing (SBST) technique is a convenient and meaningful fitness landscape. Whether one follows a targeted or a diversification driven strategy, a search landscape needs to be large, continuous, easy to construct and representative of the underlying property of interest. Constructing such a landscape is not a trivial task often requiring a significant manual effort by an expert. We present an approach for constructing meaningful and convenient fitness landscapes using neural networks (NN)-for targeted and diversification strategies alike. We suggest that output of an NN predictor can be interpreted as a fitness for a targeted strategy. The NN is trained on a corpus of execution traces and various properties of interest, prior to searching. During search, the trained NN is queried to predict an estimate of a property given an execution trace. The outputs of the NN form a convenient search space which is strongly representative of a number of properties. We believe that such a search space can be readily used for driving a search towards specific properties of interest. For a diversification strategy, we propose the use of an autoencoder; a mechanism for compacting data into an n-dimensional "latent" space. In it, datapoints are arranged according to the similarity of their salient features. We show that a latent space of execution traces possesses characteristics of a convenient search landscape: it is continuous, large and crucially, it defines a notion of similarity to arbitrary observations.

show abstract

On the Danger of Coverage Directed Test Case Generation

Cited by 55 publications

References 30 publications

From Requirements to Code: Model Based Development of a Medical Cyber Physical System

From Requirements to Code: Model Based Development of a Medical Cyber Physical System

An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing

Constructing Search Spaces for Search-Based Software Testing Using Neural Networks

Contact Info

Product

Resources

About