Static test flakiness prediction: How Far Can We Go?

Pontillo, Valeria; Palomba, Fabio; Ferrucci, Filomena

doi:10.1007/s10664-022-10227-1

Cited by 6 publications

(2 citation statements)

References 88 publications

(158 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, the rationale for using this dataset came from previous observations made by Pontillo et al. ( 2021 , 2022 ). In their study, the authors ran a state-of-the-art test smell detector named VITRuM (Pecorelli et al.…”

Section: Dataset Constructionmentioning

confidence: 99%

“…These projects were highly diverse in terms of scopes and sizes, hence representing an ideal source to mitigate possible threats to external validity-our online appendix provides detailed statistics on those projects (Pontillo et al 2023). Second, the rationale for using this dataset came from previous observations made by Pontillo et al (2021Pontillo et al ( , 2022. In their study, the authors ran a state-of-the-art test smell detector named VITRuM (Pecorelli et al 2020) and identified a high number of test smells, i.e., they found that around 80% of test cases were smelly.…”

Section: Projects Selectionmentioning

confidence: 99%

See 1 more Smart Citation

Machine learning-based test smell detection

Pontillo,

Amoroso d’Aragona,

Pecorelli

et al. 2024

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

Test smells are symptoms of sub-optimal design choices adopted when developing test cases. Previous studies have proved their harmfulness for test code maintainability and effectiveness. Therefore, researchers have been proposing automated, heuristic-based techniques to detect them. However, the performance of these detectors is still limited and dependent on tunable thresholds. We design and experiment with a novel test smell detection approach based on machine learning to detect four test smells. First, we develop the largest dataset of manually-validated test smells to enable experimentation. Afterward, we train six machine learners and assess their capabilities in within- and cross-project scenarios. Finally, we compare the ML-based approach with state-of-the-art heuristic-based techniques. The key findings of the study report a negative result. The performance of the machine learning-based detector is significantly better than heuristic-based techniques, but none of the learners able to overcome an average F-Measure of 51%. We further elaborate and discuss the reasons behind this negative result through a qualitative investigation into the current issues and challenges that prevent the appropriate detection of test smells, which allowed us to catalog the next steps that the research community may pursue to improve test smell detection techniques.

show abstract

Section: Dataset Constructionmentioning

confidence: 99%

Section: Projects Selectionmentioning

confidence: 99%

Machine learning-based test smell detection

Pontillo,

Amoroso d’Aragona,

Pecorelli

et al. 2024

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

show abstract

Static test flakiness prediction: How Far Can We Go?

2022

View full text Add to dashboard Cite

Test flakiness is a phenomenon occurring when a test case is non-deterministic and exhibits both a passing and failing behavior when run against the same code. Over the last years, the problem has been closely investigated by researchers and practitioners, who all have shown its relevance in practice. The software engineering research community has been working toward defining approaches for detecting and addressing test flakiness. Despite being quite accurate, most of these approaches rely on expensive dynamic steps, e.g., the computation of code coverage information. Consequently, they might suffer from scalability issues that possibly preclude their practical use. This limitation has been recently targeted through machine learning solutions that could predict the flakiness of tests using various features, like source code vocabulary or a mixture of static and dynamic metrics computed on individual snapshots of the system. In this paper, we aim to perform a step forward and predict test flakiness only using static metrics. We propose a large-scale experiment on 70 Java projects coming from the iDFlakies and FlakeFlagger datasets. First, we statistically assess the differences between flaky and non-flaky tests in terms of 25 test and production code metrics and smells, analyzing both their individual and combined effects. Based on the results achieved, we experiment with a machine learning approach that predicts test flakiness solely based on static features, comparing it with two state-of-the-art approaches. The key results of the study show that the static approach has performance comparable to those of the baselines. In addition, we found that the characteristics of the production code might impact the performance of the flaky test prediction models.

show abstract

Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning models

Parry¹,

Kapfhammer²,

Hilton³

et al. 2023

Empir Software Eng

View full text Add to dashboard Cite

A flaky test is a test case whose outcome changes without modification to the code of the test case or the program under test. These tests disrupt continuous integration, cause a loss of developer productivity, and limit the efficiency of testing. Many flaky test detection techniques are rerunning-based, meaning they require repeated test case executions at a considerable time cost, or are machine learning-based, and thus they are fast but offer only an approximate solution with variable detection performance. These two extremes leave developers with a stark choice. This paper introduces CANNIER, an approach for reducing the time cost of rerunning-based detection techniques by combining them with machine learning models. The empirical evaluation involving 89,668 test cases from 30 Python projects demonstrates that CANNIER can reduce the time cost of existing rerunning-based techniques by an order of magnitude while maintaining a detection performance that is significantly better than machine learning models alone. Furthermore, the comprehensive study extends existing work on machine learning-based detection and reveals a number of additional findings, including (1) the performance of machine learning models for detecting polluter test cases; (2) using the mean values of dynamic test case features from repeated measurements can slightly improve the detection performance of machine learning models; and (3) correlations between various test case features and the probability of the test case being flaky.

show abstract

Static test flakiness prediction: How Far Can We Go?

Cited by 6 publications

References 88 publications

Machine learning-based test smell detection

Machine learning-based test smell detection

Static test flakiness prediction: How Far Can We Go?

Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning models

Contact Info

Product

Resources

About