NonDex: a tool for detecting and debugging wrong assumptions on Java API specifications

Gyori, Alex; Lambeth, Ben; Shi, August; Legunsen, Owolabi; Marinov, Darko

doi:10.1145/2950290.2983932

Cited by 16 publications

(2 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Researchers presented different tools and approaches to detect flaky tests in a more efficient way. Notably, DeFlaker [28], IDFlakies [9], Shaker [10] and NonDex [29] attempt to facilitate the detection of flaky tests compared to exhaustive reruns. Because the cost of running tests is viewed as expensive, researchers also sought to suggest static alternatives for the detection.…”

Section: Related Workmentioning

confidence: 99%

FlakyCat: Predicting Flaky Tests Categories using Few-Shot Learning

Amal

Haben

Habchi

et al. 2023

2023 IEEE/ACM International Conference on Automation of Software Test (AST)

View full text Add to dashboard Cite

Flaky tests are tests that yield different outcomes when run on the same version of a program. This nondeterministic behaviour plagues continuous integration with false signals, wasting developers' time and reducing their trust in test suites. Studies highlighted the importance of keeping tests flakiness-free. Recently, the research community has been pushing towards the detection of flaky tests by suggesting many static and dynamic approaches. While promising, those approaches mainly focus on classifying tests as flaky or not and, even when high performances are reported, it remains challenging to understand the cause of flakiness. This part is crucial for researchers and developers that aim to fix it. To help with the comprehension of a given flaky test, we propose FlakyCat, the first approach to classify flaky tests based on their root cause category. FlakyCat relies on CodeBERT for code representation and leverages Siamese networks to train a multi-class classifier. We train and evaluate FlakyCat on a set of 451 flaky tests collected from open-source Java projects. Our evaluation shows that FlakyCat categorises flaky tests accurately, with an F1 score of 73%. Furthermore, we investigate the performance of our approach for each category, revealing that Async waits, Unordered collections and Time-related flaky tests are accurately classified, while Concurrency-related flaky tests are more challenging to predict. Finally, to facilitate the comprehension of FlakyCat's predictions, we present a new technique for CodeBERT-based model interpretability that highlights code statements influencing the categorization.

show abstract

Section: Related Workmentioning

confidence: 99%

FlakyCat: Predicting Flaky Tests Categories using Few-Shot Learning

Amal

Haben

Habchi

et al. 2023

2023 IEEE/ACM International Conference on Automation of Software Test (AST)

View full text Add to dashboard Cite

show abstract

“…iFixFlakies [40] aims to automatically fix order-dependent tests by utilising helper tests to reset the state required for order-dependent tests to pass. NonDex [41] targets implementation dependencies caused by assumptions developers make about under-determined APIs in the Java standard libraries, for instance the iteration order of hashed collections.…”

Section: A Flaky Test Detection Techniquesmentioning

confidence: 99%

Flaky Test Sanitisation via On-the-Fly Assumption Inference for Tests with Network Dependencies

Dietrich¹,

Rasheed²,

Tahir³

2022

Preprint

View full text Add to dashboard Cite

Flaky tests cause significant problems as they can interrupt automated build processes that rely on all tests succeeding and undermine the trustworthiness of tests. Numerous causes of test flakiness have been identified, and program analyses exist to detect such tests. Typically, these methods produce advice to developers on how to refactor tests in order to make test outcomes deterministic.We argue that one source of flakiness is the lack of assumptions that precisely describe under which circumstances a test is meaningful. We devise a sanitisation technique that can isolate flaky tests quickly by inferring such assumptions on-the-fly, allowing automated builds to proceed as flaky tests are ignored. We demonstrate this approach for Java and Groovy programs by implementing it as extensions for three popular testing frameworks (JUnit4, JUnit5 and Spock) that can transparently inject the inferred assumptions. If JUnit5 is used, those extensions can be deployed without refactoring project source code.We demonstrate and evaluate the utility of our approach using a set of six popular real-world programs, addressing known test flakiness issues in these programs caused by dependencies of tests on network availability. We find that our method effectively sanitises failures induced by network connectivity problems with high precision and recall.Index Terms-flaky tests, non-determinism, regression testing3 This refers to the number of .class files found containing methods annotated as tests, methods with @Test or @ParameterizedTest annotations 4 Standard here refers to the assumptions and assertions defined as static methods in org.junit.jupiter.api.Assumptions (JU-nit5), org.junit.Assume (JUnit4), org.junit.jupiter.api.-Assertions (JUnit5) and org.junit.Assert (JUnit4) respectively, we are counting the call sites for those methods in the respective classes.

show abstract