Developer Testing in the IDE: Patterns, Beliefs, and Behavior

Beller, Moritz; Gousios, Georgios; Panichella, Annibale; Proksch, Sebastian; Amann, Sven; Zaidman, Andy

doi:10.1109/tse.2017.2776152

Cited by 78 publications

(39 citation statements)

References 82 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, recent studies found that developers perceive and treat production code as more important than test code, thus generating quality problems in the tests [9], [10], [57], [82]. This finding is in line with the experience reported by van Deursen et al [74], who described how the quality of test code was "not as high as the production code [because] test code was not refactored as mercilessly as our production code" [74].…”

Section: Introductionsupporting

confidence: 81%

On the Relation of Test Smells to Software Code Quality

Spadini

Palomba

Zaidman

et al. 2018

2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)

Self Cite

112

108

View full text Add to dashboard Cite

Test smells are sub-optimal design choices in the implementation of test code. As reported by recent studies, their presence might not only negatively affect the comprehension of test suites but can also lead to test cases being less effective in finding bugs in production code. Although significant steps toward understanding test smells, there is still a notable absence of studies assessing their association with software quality.In this paper, we investigate the relationship between the presence of test smells and the change-and defect-proneness of test code, as well as the defect-proneness of the tested production code. To this aim, we collect data on 221 releases of ten software systems and we analyze more than a million test cases to investigate the association of six test smells and their co-occurrence with software quality. Key results of our study include:(i) tests with smells are more change-and defect-prone, (ii) 'Indirect Testing', 'Eager Test', and 'Assertion Roulette' are the most significant smells for change-proneness and, (iii) production code is more defect-prone when tested by smelly tests.

show abstract

Section: Introductionsupporting

confidence: 81%

On the Relation of Test Smells to Software Code Quality

Spadini

Palomba

Zaidman

et al. 2018

2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)

Self Cite

112

108

View full text Add to dashboard Cite

show abstract

“…After that, we use the include and exclude tag of the maven-test-plugin (or of the test task, in the case of GRADLE), so that we consider only the test cases that are actually ran when the project is built. In other words, we consider all test cases that developers of the subject systems ran when they test them, discarding those that are likely to be not reliable from the developers' perspective [60]. Once completed this filtering phase, we use 4 https://cloud.google.com/bigquery/ a pattern matching approach based on naming conventions to find the production class related to a certain test class, as done in many other previous work [34], [61], [62].…”

Section: Linking Production To Test Classesmentioning

confidence: 99%

“…When extracting test classes from the subject systems we only considered those tests available under the include tag of the MAVEN pom file. We adopted this procedure to exclude tests that are not ran when the test or package MAVEN commands are executed [60].…”

Section: Threats To Validitymentioning

confidence: 99%

Lightweight Assessment of Test-Case Effectiveness Using Source-Code-Quality Indicators

Grano

Palomba

Gall

2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Test cases are crucial to help developers preventing the introduction of software faults. Unfortunately, not all the tests are properly designed or can effectively capture faults in production code. Some measures have been defined to assess test-case effectiveness: the most relevant one is the mutation score, which highlights the quality of a test by generating the so-called mutants, ie variations of the production code that make it faulty and that the test is supposed to identify. However, previous studies revealed that mutation analysis is extremely costly and hard to use in practice. The approaches proposed by researchers so far have not been able to provide practical gains in terms of mutation testing efficiency. This leaves the problem of efficiently assessing test-case effectiveness as still open. In this paper, we investigate a novel, orthogonal, and lightweight methodology to assess test-case effectiveness: in particular, we study the feasibility to exploit production and test-code-quality indicators to estimate the mutation score of a test case. We firstly select a set of 67 factors and study their relation with test-case effectiveness. Then, we devise a mutation score estimation model exploiting such factors and investigate its performance as well as its most relevant features. The key results of the study reveal that our estimation model only based on static features has 86% of both F-Measure and AUC-ROC. This means that we can estimate the test-case effectiveness, using source-code-quality indicators, with high accuracy and without executing the tests. As a consequence, we can provide a practical approach that is beyond the typical limitations of current mutation testing techniques.Abstract-Test cases are crucial to help developers preventing the introduction of software faults. Unfortunately, not all the tests are properly designed or can effectively capture faults in production code. Some measures have been defined to assess test-case effectiveness: the most relevant one is the mutation score, which highlights the quality of a test by generating the so-called mutants, i.e., variations of the production code that make it faulty and that the test is supposed to identify. However, previous studies revealed that mutation analysis is extremely costly and hard to use in practice. The approaches proposed by researchers so far have not been able to provide practical gains in terms of mutation testing efficiency. This leaves the problem of efficiently assessing test-case effectiveness as still open. In this paper, we investigate a novel, orthogonal, and lightweight methodology to assess test-case effectiveness: in particular, we study the feasibility to exploit production and test-code-quality indicators to estimate the mutation score of a test case. We firstly select a set of 67 factors and study their relation with test-case effectiveness. Then, we devise a mutation score estimation model exploiting such factors and investigate its performance as well as its most relevant features. The key results of the s...

show abstract

“…This test suite is large, and has been written on top of a lot of human intelligence and domain knowledge [70]. Developers spend a lot of time in writing the tests [7], so that those tests exercise interesting cases (including corner cases), and so that a good oracle verifies as much as possible the program behavior.…”

Section: Introductionmentioning

confidence: 99%

A snowballing literature study on test amplification

Danglot

Vera-Pérez

et al. 2019

Journal of Systems and Software

Self Cite

View full text Add to dashboard Cite

The adoption of agile development approaches has put an increased emphasis on developer testing, resulting in software projects with strong test suites. These suites include a large number of test cases, in which developers embed knowledge about meaningful input data and expected properties in the form of oracles. This article surveys various works that aim at exploiting this knowledge in order to enhance these manually written tests with respect to an engineering goal (e.g., improve coverage of changes or increase the accuracy of fault localization). While these works rely on various techniques and address various goals, we believe they form an emerging and coherent field of research, which we call "test amplification". We devised a first set of papers from DBLP, looking for all papers containing "test" and "amplification" in their title. We reviewed the 70 papers in this set and selected the 4 papers that fit our definition of test amplification. We use these 4 papers as the seed for our snowballing study, and systematically followed the citation graph. This study is the first that draws a comprehensive picture of the different engineering goals proposed in the literature for test amplification. In particular, we note that the goal of test amplification goes far beyond maximizing coverage only. We believe that this survey will help researchers and practitioners entering this new field to understand more quickly and more deeply the intuitions, concepts and techniques used for test amplification.

show abstract

Developer Testing in the IDE: Patterns, Beliefs, and Behavior

Cited by 78 publications

References 82 publications

On the Relation of Test Smells to Software Code Quality

On the Relation of Test Smells to Software Code Quality

Lightweight Assessment of Test-Case Effectiveness Using Source-Code-Quality Indicators

A snowballing literature study on test amplification

Contact Info

Product

Resources

About