On the long-term use of visual gui testing in industrial practice: a case study

Abstract

Visual GUI Testing (VGT) is a tool-driven technique for automated GUI-based testing that uses image recognition to interact with and assert the correctness of the behavior of a system through its GUI as it is shown to the user. The technique's applicability, e.g. defect-finding ability, and feasibility, e.g. time to positive return on investment, have been shown through empirical studies in industrial practice. However, there is a lack of studies that evaluate the usefulness and challenges associated with VGT when used long-term (years) in industrial practice. This paper evaluates how VGT was adopted, applied and why it was abandoned at the music streaming application development company, Spotify, after several years of use. A qualitative study with two workshops and five well chosen employees is performed at the company, supported by a survey, which is analyzed with a grounded theory approach to answer the study's three research questions. The interviews provide insights into the challenges, problems and limitations, but also benefits, that Spotify experienced during the adoption and use of VGT. However, due to the technique's drawbacks, VGT has been abandoned for a new technique/framework, simply called the Test interface. The Test interface is considered more robust and flexible for Spotify's needs but has several drawbacks, including that it does not test the actual GUI as shown to the user like VGT does. From the study's results it is concluded that VGT can be used long-term in industrial practice but it requires organizational change as well as engineering best practices to be beneficial. Through synthesis of the study's results, and results from previous work, a set of guidelines are presented that aim to aid practitioners to adopt and use VGT in industrial practice. However, due to the abandonment of the technique, future research is required to analyze in what types of projects the technique is, and is not, long-term viable. To this end, we also present Spotify's Test interface solution for automated GUI-based testing and conclude that it has its own benefits and drawbacks.

Knowledge Capture and Reuse Through Expert’s Activity Monitoring in Engineering Design

“…They go on to classify following three categories: pixel coordinate approaches, event interception methods and computer vision paradigms. (Qureshi and Nadeem 2013) propose an extensive analysis of approaches falling into the second generation identified by (Alégroth and Feldt 2017). The interesting point of this is that it illustrates the maturity of this field in contrast to the third generation which is still emerging at the time.…”

“…Although this does present some challenges of its own (especially when the notion of acceptable is fuzzy), it certainly misses the point of testing the GUI layers. (Alégroth and Feldt 2017) seek new solutions to this problem, and start by establishing a clean taxonomy of the existing approaches they have collected from existing tools and existing academic works, focusing on validating the GUI layers of software systems. They identify three distinctive steps in the history of the art of software testing.…”

Estimating Return on Investment for GUI Test Automation Frameworks

“…In order to spare human testers of the tedious, laborious and error-prone work of manually interacting with the GUI of a SUT (also referred as Manual GUI Testing-MGT), various techniques and tools that enable automated GUI testing have been proposed in literature [13,14] and practice [8,15,16]. Even though Automated GUI tests (AGT) are more reliable, reproducible, and tolerant to small GUI related changes in the SUT [17], they demand maintenance of the test suite to match the new GUI [8].…”

Automated reporting of GUI design violations for mobile apps

“…Visual GUI Testing: A concept known as Visual GUI Testing (VGT) aims to test certain visual aspects of a software application's GUI as well as the underlying functional properties. To accomplish this visual GUI testing usually executes actions on a target applications in order to exercise app functionality [13,14,23,38]. In contrast to these approaches, Gvt is designed to apply to mobile-specific DVs, is tailored for the mock-up driven development practice, and is aimed only at verifying visual properties of a mobile app's GUI.…”