Is 100% Test Coverage a Reasonable Requirement? Lessons Learned from a Space Software Project

Prause, Christian R.; Werner, J.; Hornig, Kay; Bosecker, Sascha; Kuhrmann, Marco

doi:10.1007/978-3-319-69926-4_25

Cited by 13 publications

(4 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…coverage above 80%, or relatively, e.g., coverage value within ϵ% of the coverage achieved by the best performing technique. As there is no absolute definition of what percentage of coverage would be considered good [58], we compare the performance of portfolio techniques relatively.…”

Section: Performance Space Ymentioning

confidence: 99%

Instance Space Analysis of Search-Based Software Testing

Neelofar

Smith‐Miles

Muñoz

et al. 2023

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Search-based software testing (SBST) is now a mature area, with numerous techniques developed to tackle the challenging task of software testing. SBST techniques have shown promising results and have been successfully applied in the industry to automatically generate test cases for large and complex software systems. Their effectiveness, however, has been shown to be problem dependent. In this paper, we revisit the problem of objective performance evaluation of SBST techniques in light of recent methodological advances -in the form of Instance Space Analysis (ISA) -enabling the strengths and weaknesses of SBST techniques to be visualised and assessed across the broadest possible space of problem instances (software classes) from common benchmark datasets. We identify features of SBST problems that explain why a particular instance is hard for an SBST technique, reveal areas of hard and easy problems in the instance space of existing benchmark datasets, and identify the strengths and weaknesses of state-of-the-art SBST techniques. In addition, we examine the diversity and quality of common benchmark datasets used in experimental evaluations.

show abstract

Section: Performance Space Ymentioning

confidence: 99%

Instance Space Analysis of Search-Based Software Testing

Neelofar

Smith‐Miles

Muñoz

et al. 2023

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…Code coverage is a general test suite reliability metric, the history of code coverage dates back to 1963 [43]. Software engineers have proposed different types of coverage criteria such as line coverage, branch coverage, path coverage, and modified condition/decision coverage [6,54,61]. Mutation testing is another widely-used metric for measuring the reliability of a test suite.…”

Section: Related Workmentioning

confidence: 99%

Who Judges the Judge: An Empirical Study on Online Judge Tests

Liu

Yan

Zhang

et al. 2023

Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

Online Judge platforms play a pivotal role in education, competitive programming, recruitment, career training, and large language model training. They rely on predefined test suites to judge the correctness of submitted solutions. It is therefore important that the solution judgement is reliable and free from potentially misleading false positives (i.e., incorrect solutions that are judged as correct). In this paper, we conduct an empirical study of 939 coding problems with 541,552 solutions, all of which are judged to be correct according to the test suites used by the platform, finding that 43.1% of the problems include false positive solutions (3,700 bugs are revealed in total). We also find that test suites are, nevertheless, of high quality according to widely-studied test effectiveness measurements: 88.2% of false positives have perfect (100%) line coverage, 78.9% have perfect branch coverage, and 32.5% have a perfect mutation score. Our findings indicate that more work is required to weed out false positive solutions and to further improve test suite effectiveness. We have released the detected false positive solutions and the generated test inputs to facilitate future research. CCS CONCEPTS• Software and its engineering → Software testing and debugging.

show abstract

“…Also in the Space domain, the European Cooperation for Space Standardization (ECSS) standards 17 provide guidelines and examples of software metrics. These metrics can be used in space system development with respect to the requirements 18 and to provide a coherent view of the software metrication program definition and implementation. In this article, we also use the ECSS standard to fill gaps in the details of metrics obtained from the systematic review (see also Section 3.3.2).…”

Section: Background and Related Workmentioning

confidence: 99%

Metrics in automotive software development: A systematic literature review

Vogel

Knapik

Cohrs

et al. 2020

J Software Evolu Process

Self Cite

View full text Add to dashboard Cite

Software is an integrated part of new features within the automotive sector, car manufacturers, the Hersteller Initiative Software (HIS) consortium defined metrics to determine software quality. Yet, problems with assigning metrics to quality attributes often occur in practice. The specified boundary values lead to discussions between contractors and clients as different standards and metric sets are used. This paper studies metrics used in the automotive sector and the quality attributes they address. The HIS, ISO/IEC 25010:2011, and ISO/IEC 26262:2018 are utilized to draw a big picture illustrating (i) which metrics and boundary values are reported in literature, (ii) how the metrics match the standards, (iii) which quality attributes are addressed, and (iv) how the metrics are supported by tools. Our findings from analyzing 38 papers include a catalog of 112 metrics of which 17 define boundary values and 48 are supported by tools. Most of the metrics are concerned with source code, are generic, and not specifically designed for automotive software development. We conclude that many metrics exist, but a clear definition of the metrics' context, notably regarding the construction of flexible and efficient measurement suites, is missing.

show abstract

Is 100% Test Coverage a Reasonable Requirement? Lessons Learned from a Space Software Project

Cited by 13 publications

References 11 publications

Instance Space Analysis of Search-Based Software Testing

Instance Space Analysis of Search-Based Software Testing

Who Judges the Judge: An Empirical Study on Online Judge Tests

Metrics in automotive software development: A systematic literature review

Contact Info

Product

Resources

About