Toward static test flakiness prediction: a feasibility study

Pontillo, Valeria; Palomba, Fabio; Ferrucci, Filomena

doi:10.1145/3472674.3473981

Cited by 13 publications

(8 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The promising results achieved by our previous work (Pontillo et al 2021) indicated the feasibility of devising a static approach to flaky test prediction. Hence, in this paper, first we extend our preliminary work by replicating the initial analyses on the FLAKEFLAGGER dataset, 2 in an effort of increasing the generalizability of our results.…”

Section: Introductionmentioning

confidence: 82%

“…With respect to the studies discussed above and the results obtained from our previous feasibility study (Pontillo et al 2021), our work can be considered as complementary, since it contributes with an additional technique to predict test flakiness that only considers static metrics. It is important to emphasize that our research is driven by a key consideration: a prediction only based on static metrics could lead to benefits in terms of (1) computational costs, as it would avoid the computation of dynamic metrics that would require the execution of the entire test suite; (2) interpretability, as it would allow developers to focus on a refactoring of test cases guided by the static metrics and smells that impact more the likelihood of the test becoming flaky.…”

Section: Related Workmentioning

confidence: 97%

“…To face the scalability limitations of the currently available techniques, our previous work (Pontillo et al 2021) aimed at conducting a feasibility study to assess whether a static prediction of test flakiness would be possible, i.e., whether we could identify likely flaky test cases only based on their design. In particular, we took into account the IDFLAKIES dataset, 1 and investigated the differences between flaky and non-flaky tests in terms of 25 test and production code metrics and smells.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Static test flakiness prediction: How Far Can We Go?

2022

Self Cite

View full text Add to dashboard Cite

Test flakiness is a phenomenon occurring when a test case is non-deterministic and exhibits both a passing and failing behavior when run against the same code. Over the last years, the problem has been closely investigated by researchers and practitioners, who all have shown its relevance in practice. The software engineering research community has been working toward defining approaches for detecting and addressing test flakiness. Despite being quite accurate, most of these approaches rely on expensive dynamic steps, e.g., the computation of code coverage information. Consequently, they might suffer from scalability issues that possibly preclude their practical use. This limitation has been recently targeted through machine learning solutions that could predict the flakiness of tests using various features, like source code vocabulary or a mixture of static and dynamic metrics computed on individual snapshots of the system. In this paper, we aim to perform a step forward and predict test flakiness only using static metrics. We propose a large-scale experiment on 70 Java projects coming from the iDFlakies and FlakeFlagger datasets. First, we statistically assess the differences between flaky and non-flaky tests in terms of 25 test and production code metrics and smells, analyzing both their individual and combined effects. Based on the results achieved, we experiment with a machine learning approach that predicts test flakiness solely based on static features, comparing it with two state-of-the-art approaches. The key results of the study show that the static approach has performance comparable to those of the baselines. In addition, we found that the characteristics of the production code might impact the performance of the flaky test prediction models.

show abstract

Section: Introductionmentioning

confidence: 82%

Section: Related Workmentioning

confidence: 97%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Static test flakiness prediction: How Far Can We Go?

2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…These projects were highly diverse in terms of scopes and sizes, hence representing an ideal source to mitigate possible threats to external validity-our online appendix provides detailed statistics on those projects (Pontillo et al 2023). Second, the rationale for using this dataset came from previous observations made by Pontillo et al (2021Pontillo et al ( , 2022. In their study, the authors ran a state-of-the-art test smell detector named VITRuM (Pecorelli et al 2020) and identified a high number of test smells, i.e., they found that around 80% of test cases were smelly.…”

Section: Projects Selectionmentioning

confidence: 99%

Machine learning-based test smell detection

Pontillo,

Amoroso d’Aragona,

Pecorelli

et al. 2024

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

Test smells are symptoms of sub-optimal design choices adopted when developing test cases. Previous studies have proved their harmfulness for test code maintainability and effectiveness. Therefore, researchers have been proposing automated, heuristic-based techniques to detect them. However, the performance of these detectors is still limited and dependent on tunable thresholds. We design and experiment with a novel test smell detection approach based on machine learning to detect four test smells. First, we develop the largest dataset of manually-validated test smells to enable experimentation. Afterward, we train six machine learners and assess their capabilities in within- and cross-project scenarios. Finally, we compare the ML-based approach with state-of-the-art heuristic-based techniques. The key findings of the study report a negative result. The performance of the machine learning-based detector is significantly better than heuristic-based techniques, but none of the learners able to overcome an average F-Measure of 51%. We further elaborate and discuss the reasons behind this negative result through a qualitative investigation into the current issues and challenges that prevent the appropriate detection of test smells, which allowed us to catalog the next steps that the research community may pursue to improve test smell detection techniques.

show abstract

“…In a broader sense, detection involves the process of discovery. Video images [1], acoustic signals [2], radio signals [3], or even smell [4] can be used for detection and localization. This review focuses on the methods of detecting and localizing sound sources-that is, the acoustic signal.…”

Section: Introductionmentioning

confidence: 99%

A Survey of Sound Source Localization and Detection Methods and Their Applications

Jekateryńczuk,

Piotrowski

2023

Sensors

View full text Add to dashboard Cite

This study is a survey of sound source localization and detection methods. The study provides a detailed classification of the methods used in the fields of science mentioned above. It classifies sound source localization systems based on criteria found in the literature. Moreover, an analysis of classic methods based on the propagation model and methods based on machine learning and deep learning techniques has been carried out. Attention has been paid to providing the most detailed information on the possibility of using physical phenomena, mathematical relationships, and artificial intelligence to determine sound source localization. Additionally, the article underscores the significance of these methods within both military and civil contexts. The study culminates with a discussion of forthcoming trends in the realms of acoustic detection and localization. The primary objective of this research is to serve as a valuable resource for selecting the most suitable approach within this domain.

show abstract

Toward static test flakiness prediction: a feasibility study

Cited by 13 publications

References 32 publications

Static test flakiness prediction: How Far Can We Go?

Static test flakiness prediction: How Far Can We Go?

Machine learning-based test smell detection

A Survey of Sound Source Localization and Detection Methods and Their Applications

Contact Info

Product

Resources

About