An Empirical Study of Flaky Tests in Android Apps

Thorve, Swapna; Sreshtha, Chandani; Meng, Na

doi:10.1109/icsme.2018.00062

Cited by 54 publications

(40 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, we attempt to predict flakiness using not only the vocabulary of test code (like Pinto et al did) but also the Code Under Test (CUT). This endeavour follows the findings of many reports [1], [16], [17] revealing that much flakiness manifests in the CUT. Hence, we conduct a comparative study that highlights the impact of the two feature sets, i.e., sources of vocabulary on flakiness prediction.…”

Section: Introductionsupporting

confidence: 78%

A Replication Study on the Usability of Code Vocabulary in Predicting Flaky Tests

Haben

Habchi

Papadakis

et al. 2021

2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)

View full text Add to dashboard Cite

Industrial reports indicate that flaky tests are one of the primary concerns of software testing mainly due to the false signals they provide. To deal with this issue, researchers have developed tools and techniques aiming at (automatically) identifying flaky tests with encouraging results. However, to reach industrial adoption and practice, these techniques need to be replicated and evaluated extensively on multiple datasets, occasions and settings. In view of this, we perform a replication study of a recently proposed method that predicts flaky tests based on their vocabulary. We thus replicate the original study on three different dimensions. First, we replicate the approach on the same subjects as in the original study but using a different evaluation methodology, i.e., we adopt a time-sensitive selection of training and test sets to better reflect the envisioned use case. Second, we consolidate the findings of the initial study by building a new dataset of 837 flaky tests from 9 projects in a different programming language, i.e., Python while the original study was in Java, which comforts the generalisability of the results. Third, we propose an extension to the original approach by experimenting with different features extracted from the Code Under Test. We find that a more robust validation consistently decreases performance on the reported results of the original study, but, fortunately, the model remains capable to decently predict flaky tests. We find re-assuring results that the vocabulary-based models can also be used to predict test flakiness in Python. Finally, we find that the information lying in the Code Under Test has a limited impact on the performance of the vocabulary-based models.

show abstract

Section: Introductionsupporting

confidence: 78%

A Replication Study on the Usability of Code Vocabulary in Predicting Flaky Tests

Haben

Habchi

Papadakis

et al. 2021

2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)

View full text Add to dashboard Cite

show abstract

“…In fact, considering the survey results garnered in References 27,34,47,51,52, we are envisioning that future works in this area should somehow depend on ML, deep learning (DL), or hybrid techniques.…”

Section: Results and Analysismentioning

confidence: 99%

“…In addition to the general sources of flakiness, some domain‐specific sources have been reported in the literature. For example, research was reported in Reference 51 that focuses on detecting flaky tests in Android. The authors of this article detected 77 flaky commits among 29 GitHub Android projects.…”

Section: Results and Analysismentioning

confidence: 99%

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

et al. 2020

View full text Add to dashboard Cite

A flaky test is a test that may lead to different results in different runs on a single code under test without any change in the test code. Test flakiness is a noxious phenomenon that slows down software deployment, and increases the expenditures in a broad spectrum of platforms such as software‐defined networks and Internet of Things environments. Industrial institutes and labs have conducted a whole lot of research projects aiming at tackling this problem. Although this issue has been receiving more attention from academia in recent years, the academic research community is still behind the industry in this area. A systematic review and trend analysis on the existing approaches for detecting and root causing flaky tests can pave the way for future research on this topic. This can help academia keep pace with industrial advancements and even lead the research in this field. This article first presents a comprehensive review of recent achievements of the industry as well as academia regarding the detection and mitigation of flaky tests. In the next step, recent trends in this line of research are analyzed and a roadmap is established for future research.

show abstract

“…3 Many empirical studies have been conducted aiming at better understanding the nature and extent of test flakiness (e.g., [1], [9], [23]- [31]). Both studies by Luo et al [1] and Vahabzadeh et al [24] examined the causes of flaky tests over the central repository of the Apache Software Foundation, whereas Thorve et al [26] conducted a similar study over Android apps. Other studies aim at assessing the relevance and cost of the problem, including the work by Labuschagne et al [9] over 61 projects from GitHub Archive, and the one by Rahman and Rigby [25] who studied how many of the crash reports submitted by Firefox users were associated with flaky tests, while Eck et al [28] asked developers from Mozilla to classify previously fixed flaky tests.…”

Section: Related Workmentioning

confidence: 99%

Know You Neighbor: Fast Static Prediction of Test Flakiness

et al. 2021

View full text Add to dashboard Cite

Context: Flaky tests plague regression testing in Continuous Integration environments by slowing down change releases and wasting testing time and effort. Despite the growing interest in mitigating the burden of test flakiness, how to efficiently and effectively detect flaky tests is still an open problem.Objective: In this study, we present and evaluate FLAST, an approach designed to statically predict test flakiness. FLAST leverages vector-space modeling, similarity search, dimensionality reduction, and k-Nearest Neighbor classification in order to timely and efficiently detect test flakiness. Method: In order to gain insights into the efficiency and effectiveness of FLAST, we conduct an empirical evaluation of the approach by considering 13 real-world projects, for a total of 1,383 flaky and 26,702 nonflaky tests. We carry out a quantitative comparison of FLAST with the state-of-the-art methods to detect test flakiness, by considering a balanced dataset comprising 1,402 real-world flaky and as many non-flaky tests. Results: From the results we observe that the effectiveness of FLAST is comparable with the state-of-theart, while providing considerable gains in terms of efficiency. In addition, the results demonstrate how by tuning the threshold of the approach FLAST can be made more conservative, so to reduce false positives, at the cost of missing more potentially flaky tests. Conclusion:The collected results demonstrate that FLAST provides a fast, low-cost and reliable approach that can be used to guide test rerunning, or to gate the inclusion of new potentially flaky tests.

show abstract

An Empirical Study of Flaky Tests in Android Apps

Cited by 54 publications

References 4 publications

A Replication Study on the Usability of Code Vocabulary in Predicting Flaky Tests

A Replication Study on the Usability of Code Vocabulary in Predicting Flaky Tests

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

Know You Neighbor: Fast Static Prediction of Test Flakiness

Contact Info

Product

Resources

About