The Impact of Coverage on Bug Density in a Large Industrial Software Project

Bach, Thomas; Andrzejak, Artur; Pannemans, Ralf; Lo, David

doi:10.1109/esem.2017.44

Cited by 15 publications

(8 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These results show that there may be a misalignment between the SOP and tested software parts, providing an answer to question RQ 3 . Bach et al (2017) investigated the relationship between the coverage provided by a test suite and its effectiveness. The approach adopted in Bach et al (2017) can also be used as another strategy to get evidence of the possible mismatch between SOP and the tested software parts, as well as the relation between this misalignment and software faults.…”

Section: Exs-at 2 : Resultsmentioning

confidence: 99%

“…Bach et al (2017) investigated the relationship between the coverage provided by a test suite and its effectiveness. The approach adopted in Bach et al (2017) can also be used as another strategy to get evidence of the possible mismatch between SOP and the tested software parts, as well as the relation between this misalignment and software faults. The approach used in Bach et al (2017) defines two scenarios referring to the hypothesis investigated:…”

Section: Exs-at 2 : Resultsmentioning

confidence: 99%

“…A high coverage rate can reduce the volume of future bugs. Bach et al (2017) analyzed identified faults using the failures reported by software users and the relation of the data obtained by this analysis to the coverage provided by the test suite of the respective software.…”

Section: Exs-at 3 : Failures In Untested Sop Partsmentioning

confidence: 99%

See 2 more Smart Citations

Software Operational Profile vs. Test Profile: Towards a Better Software Testing Strategy

Júnior

Morimoto

Fabbri

et al. 2020

JSERD

View full text Add to dashboard Cite

Software Operational Profile (SOP) is a software specification based on how users use the software. This specification corresponds to a quantitative representation of software that identifies the most used software parts. As software reliability depends on the context in which users operate the software, the SOP is used in software reliability engineering. However, there are evidences of a misalignment between the software tested parts and SOP. Therefore, this paper investigates a possible misalignment between SOP and the tested software parts to obtain, based on experimental data, more evidence of this misalignment. We performed an exploratory study composed of four activities to verify: a) whether there are significant variations in how users operate the software; b) whether there is a misalignment between SOP and the tested software parts; c) if failures occur in untested SOP parts in case of misalignment; d) in case of misalignment between SOP and untested software parts, whether a test strategy based on the amplification of the existent test set with additional test data generated automatically, can contribute to reduce the misalignment. We collected data form four software while users were operating them. We analyzed this collected data in an attempt to reach the goals of this work. To evaluate the originality of this research, we performed a Literature Systematic Review (SLR) and presented its conclusions. The obtained results evidence that there are significant variations in how users operate the software and also that there is a misalignment between SOP and the tested software parts when we evaluated the four software mentioned above. There are also indications of the occurrence of failures in the untested SOP parts. Although the test strategy mentioned above has reduced the possible misalignment, the test strategy is not enough to avoid it, thus denoting the need of specifics test strategies using SOP as a test criterion. These results indicate that SOP becomes relevant not only to software reliability engineering but also to contribute to testing activities, regardless of the adopted strategy.

show abstract

Section: Exs-at 2 : Resultsmentioning

confidence: 99%

Section: Exs-at 2 : Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Software Operational Profile vs. Test Profile: Towards a Better Software Testing Strategy

Júnior

Morimoto

Fabbri

et al. 2020

JSERD

View full text Add to dashboard Cite

show abstract

“…30% threshold resulted in the right balance for further statistical analysis. To answer RQ2 we used an approach similar to binary testedness previously reported by Ahmed et al [2] and Bach et al [5]. Binary testedness separates source code in two (binary) groups.…”

Section: Methodsmentioning

confidence: 99%

Zones of Pain: Visualising the Relationship Between Software Architecture and Defects

Petrić

Hall

Bowes

2020

Communications in Computer and Information Science

View full text Add to dashboard Cite

Substantial development time is devoted to software maintenance and testing. As development resources are usually finite, there is a risk that some components receive insufficient effort for thorough testing. Architectural complexity (e.g. tight coupling) can make effective testing particularly challenging. Software components with high architectural complexity are more likely be defect-prone. The aim of this study is to investigate the relationship between established architectural attributes and defect-proneness. We used the architectural attributes: abstractness, instability and distance to measure the architectural complexity of software components. We investigated the ability of these attributes to discriminate between defective and non-defective components on four open source systems. We visualised defect-proneness by plotting architectural complexity and defectiveness using Martin's 'Zones of Pain'. Our results show that architecture has an inconsistent impact on defectproneness. Some architecturally complex components seem immune to defects in some projects. In other projects architecturally complex components significantly suffer from defects. Where architectural complexity does increase defect-proneness the impact is strong. We recommend practitioners monitor the architectural complexity of their software components over time by visualising potential defect-proneness using Martin's Zones of Pain.

show abstract

“…If a test suite is effective, we should expect fewer defects in the covered as opposed to non-covered code. Bach et al analysed large industrial software to investigate whether test covered code is related to fewer defect-fixes compared to noncovered code [5]. Bach et al report that the distribution of defects in covered versus non-covered code is not uniform, and that more defects appear in code which has not been tested.…”

Section: Introductionmentioning

confidence: 99%

How Effectively Is Defective Code Actually Tested?

Petrić

Hall

Bowes

2018

Proceedings of the 14th International Conference on Predictive Models and Data Analytics in Software Engineering

View full text Add to dashboard Cite

Background: Newspaper headlines still regularly report latent software defects. Such defects have often evaded testing for many years. It remains difficult to identify how well a system has been tested. It also remains difficult to assess how successful at finding defects particular tests are. Coverage and mutation testing are frequently used to asses test effectiveness. We look more deeply at the performance of commonly used JUnit testing by assessing how much JUnit testing was done and how effective that testing was at detecting defects in seven open source systems. Aim: We aim to identify whether defective code has been effectively tested by JUnit tests as non-defective code. We also aim to identify the characteristics of JUnit tests that are related to identifying defects. Methodology: We first extract the defects from seven open source projects using the SZZ algorithm. We match those defects with JUnit tests to identify the proportion of defects that were covered by JUnit tests. We also do the same for non-defective code. We then use Principal Component Analysis and machine learning to investigate the characteristics of JUnit tests that were successful in identifying defects. Results: Our findings suggest that most of the open source systems we investigated are under-tested. On average over 66% of defective methods were not linked to any JUnit tests. We show that the number of methods touched by a JUnit test is strongly related to that test uncovering a defect. Conclusion: More JUnit tests need to be produced for the seven open source systems that we investigate. JUnit tests need to be relatively sophisticated, in particular they should touch more than just one method during the test.

show abstract

The Impact of Coverage on Bug Density in a Large Industrial Software Project

Cited by 15 publications

References 12 publications

Software Operational Profile vs. Test Profile: Towards a Better Software Testing Strategy

Software Operational Profile vs. Test Profile: Towards a Better Software Testing Strategy

Zones of Pain: Visualising the Relationship Between Software Architecture and Defects

How Effectively Is Defective Code Actually Tested?

Contact Info

Product

Resources

About