The Art of Testing Less without Sacrificing Quality

Herzig, Kim; Greiler, Michaela; Czerwonka, Jacek; Murphy, Brendan

doi:10.1109/icse.2015.66

Cited by 66 publications

(45 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Cost information: Some TCP algorithms take into consideration the cost of each test case to prioritize the test cases, e.g. execute all test cases in ascending order of their runtime estimated from the execution history [23]. • Test description information: Standard natural language processing techniques (stopword removal, stemming, bag-of-word, etc.)…”

Section: What Information Is Used Different Tcp Algorithms Utilize Dmentioning

confidence: 99%

TERMINATOR: better automated UI test case prioritization

Fahid

Rothermel

et al. 2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

View full text Add to dashboard Cite

Automated UI testing is an important component of the continuous integration process of software development. A modern web-based UI is an amalgam of reports from dozens of microservices written by multiple teams. Queries on a page that opens up another will fail if any of that page's microservices fails. As a result, the overall cost for automated UI testing is high since the UI elements cannot be tested in isolation. For example, the entire automated UI testing suite at LexisNexis takes around 30 hours (3-5 hours on the cloud) to execute, which slows down the continuous integration process.To mitigate this problem and give developers faster feedback on their code, test case prioritization techniques are used to reorder the automated UI test cases so that more failures can be detected earlier. Given that much of the automated UI testing is "black box" in nature, very little information (only the test case descriptions and testing results) can be utilized to prioritize these automated UI test cases. Hence, this paper evaluates 17 "black box" test case prioritization approaches that do not rely on source code information. Among these, we propose a novel TCP approach, that dynamically re-prioritizes the test cases when new failures are detected, by applying and adapting a state of the art framework from the total recall problem. Experimental results on LexisNexis automated UI testing data show that our new approach (which we call TERMINATOR), outperformed prior state of the art approaches in terms of failure detection rates with negligible CPU overhead. CCS CONCEPTS• Software and its engineering → Software testing and debugging; • Information systems → Learning to rank.

show abstract

Section: What Information Is Used Different Tcp Algorithms Utilize Dmentioning

confidence: 99%

TERMINATOR: better automated UI test case prioritization

Fahid

Rothermel

et al. 2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

View full text Add to dashboard Cite

show abstract

“…Several researchers have proposed the use of model‐based techniques for regression test selection. Another approach has been proposed to dynamically skip tests when the expected cost of running the test exceeds the expected cost of removing the test . Korel et al used an extended finite‐state machine model dependence analysis for identifying modifications .…”

Section: Related Workmentioning

confidence: 99%

“…Another approach has been proposed to dynamically skip tests when the expected cost of running the test exceeds the expected cost of removing the test. 30 Korel et al used an extended finite-state machine model dependence analysis for identifying modifications. 31 Andrews et al applied model-based selective regression testing to a railroad crossing control system based on the behavioral model of a system and a behavioral test suite.…”

Section: Related Workmentioning

confidence: 99%

A learning algorithm for optimizing continuous integration development and testing practice

2018

View full text Add to dashboard Cite

Summary Continuous integration, at its core, includes a set of practices that aim to prevent and reduce the cost of software integration issues by merging working software copies often. Regression testing is considered a good practice in software development with continuous integration, which ensures that code changes are not negatively affecting software functionality. As, nowadays, software development is carried out iteratively, with small code increments continuously developed and regression tested, it is of critical importance that continuous regression testing is time efficient. However, in practice, regression testing is often long lasting and faces scalability problems as software grows larger or as software changes are made more frequently. One contributing factor to these issues is test redundancy, which causes the same software functionality being tested multiple times across a test suite. In large‐scale software, especially highly configurable software, redundancy in continuous regression testing can significantly grow the size of test suites and negatively affect the cost effectiveness of continuous integration. This paper presents a practical learning algorithm for optimizing continuous integration testing by reducing ineffective test redundancy in regression suites. The novelty of the algorithm lies in learning and predicting the fault‐detection effectiveness of continuous integration tests using historical test records and combining this information with coverage‐based redundancy metrics. The goal is to identify ineffective redundancy, which is maximally reduced in the resulting regression test suite, thus reducing test time and improving the performance of continuous integration. We apply and evaluate the algorithm in two industrial projects of continuous integration. The results show that the proposed algorithm can improve the efficiency of continuous integration practice in terms of decreasing test execution time by 38% on average compared to the industry practice of our case study and by 40% on average compared to the retest‐all approach. The results further demonstrate no significant reduction in fault‐detection effectiveness of continuous regression testing. This suggests that the proposed algorithm contributes to the state of the practice in the continuous integration development and testing of highly configurable systems.

show abstract

“…Recent examples include work by Herzig et al [9] and Mondal et al [14], where the goal has been to optimize on cost, code coverage, test diversity and/or test duration.…”

Section: Related Workmentioning

confidence: 99%

“…What advantages and disadvantages come with this approach compared to a mathematical optimization approach such as the one investigated in [9], or approaches including a cost-benefit ratio analysis such as the one investigated in [14]?…”

Section: Other Approaches For Priority Mergingmentioning

confidence: 99%

Experience Report: Automated System Level Regression Test Prioritization Using Multiple Factors

Strandberg

Sundmark

Afzal

et al. 2018

EasyChair Preprints

View full text Add to dashboard Cite

Abstract-We propose a new method of determining an effective ordering of regression test cases, and describe its implementation as an automated tool called SuiteBuilder developed by Westermo Research and Development AB. The tool generates an efficient order to run the cases in an existing test suite by using expected or observed test duration and combining priorities of multiple factors associated with test cases, including previous fault detection success, interval since last executed, and modifications to the code tested. The method and tool were developed to address problems in the traditional process of regression testing, such as lack of time to run a complete regression suite, failure to detect bugs in time, and tests that are repeatedly omitted. The tool has been integrated into the existing nightly test framework for Westermo software that runs on largescale data communication systems. In experimental evaluation of the tool, we found significant improvement in regression testing results. The re-ordered test suites finish within the available time, the majority of fault-detecting test cases are located in the first third of the suite, no important test case is omitted, and the necessity for manual work on the suites is greatly reduced.

show abstract

The Art of Testing Less without Sacrificing Quality

Cited by 66 publications

References 28 publications

TERMINATOR: better automated UI test case prioritization

TERMINATOR: better automated UI test case prioritization

A learning algorithm for optimizing continuous integration development and testing practice

Experience Report: Automated System Level Regression Test Prioritization Using Multiple Factors

Contact Info

Product

Resources

About