Automated testing of GUI applications: Models, tools, and controlling flakiness

Abstract

Abstract-System testing of applications with graphical user interfaces (GUIs) such as web browsers, desktop, or mobile apps, is more complex than testing from the command line. Specialized tools are needed to generate and run test cases, models are needed to quantify behavioral coverage, and changes in the environment, such as the operating system, virtual machine or system load, as well as starting states of the executions, impact the repeatability of the outcome of tests making tests appear flaky.In this tutorial, we present an overview of the state of the art in GUI testing, consisting of both lectures and demonstrations on various platforms (desktop, web and mobile applications), using an open source testing tool, GUITAR. We show how to setup a system under test, how to extract models without source code, and how to then use those models to generate and replay test cases. We then present a lecture on the various factors that may cause flakiness in the execution of GUI-centric software, and hence impact the results of analyses and experiments based on such software. We end with a demonstration of a community resource for sharing GUI testing artifacts aimed at controlling these factors.This tutorial targets both researchers who develop techniques for testing GUI software, and practitioners from industry who want to learn more about model-based GUI testing or who run and rerun GUI tests and often find their runs are flaky. I. TUTORIAL OVERVIEWSystem testing of applications on which the user interacts through a graphical user interface (GUI) such as common desktop applications, web clients or mobile apps, requires a more complex test harness than traditional command line interfaces. This event-driven paradigm creates several challenges. Models are needed to evaluate behavioral coverage, and specialized tools are required to generate and replay tests in an automated fashion [1]. And these tools and techniques may be platform dependent. Test cases are sequences of events, instead of simple inputs; actions that are performed on the interface in a specific order. The starting state of the application and its environment (e.g. Java version, OS and system load) can both impact the outcome of a test, and as events are executed, threads may be spawned to perform calculations, which will change the system state, and may incur delays before they are complete. All of these factors, if not carefully controlled lead to flakiness -the inability to repeat execution in a reliable manner, and can affect both test results and experimentation on new techniques. This tutorial aims to de-mystify testing in event-driven systems by presenting an overview on the state of the art in GUI testing, with lectures on modeling, test generation and replay, as well as a discussion of the important factors that should be considered for controlling flakiness. We include demonstrations on the basics of automated GUI testing using the open source framework GUITAR [2], with examples from various domains such as desktop, web and mobile applications. ...

Notice of Retraction: Does Refactoring of Test Smells Induce Fixing Flaky Tests?

“…Memon and Cohen [20] described a set of negative effects that flaky tests may create during regression testing, finding that their presence can even lead to missing deadlines due to the fact that certain features cannot be tested sufficiently [20]. Marinescu et al [63] analyzed the evolution of test suite coverage, reporting that the presence of flaky tests produces an intermittent variation of the branch coverage.…”

“…As pointed out by several researchers and practitioners, flaky tests represent an important issue for regression testing [18], [25], [14], [20], [21], [22]. Memon and Cohen [20] described a set of negative effects that flaky tests may create during regression testing, finding that their presence can even lead to missing deadlines due to the fact that certain features cannot be tested sufficiently [20].…”

“…Perhaps most importantly, from a psychological point of view flaky tests can reduce a developer's confidence in the tests, possibly leading to ignoring actual test failures [17]. Because of this, the research community has spent considerably effort on trying to understand the causes behind test flakiness [18], [20], [21], [22] and on devising automated techniques able to fix flaky tests [23], [24], [25]. However, most of this research mainly focused the attention on some specific causes possibly leading to the introduction of flaky tests, such as concurrency [26], [25], [27] or test order dependency [22] issues, thus proposing ad-hoc solutions that cannot be used to fix flaky tests characterized by other root causes.…”

On Building Test Automation System for Mobile Applications Using GUI Ripping

“…This crawler will then reveal the fault points in the application such as runtime crashes and it can also be used in the regression testing. GUI ripping has become a popular technology in GUI test automation using well-defined GUI ripping models [2,3]. GUI-based testing has been discussed in numerous papers.…”

“…An accessibility framework known as Microsoft UI Automation (UIA) is used to perform Reverse engineering to generate the event flow graph [1]. Memon et al discussed GUI test modeling, test generation and replay, and factors for controlling flakiness of generated test cases [2]. The models used for event spacing are state machine and graph models.…”