Maintenance of automated test suites in industry: An empirical study on Visual GUI Testing

Alégroth, Emil; Feldt, Robert; Kolström, Pirjo

doi:10.1016/j.infsof.2016.01.012

Cited by 50 publications

(41 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A more recently published study has also provided support regarding the feasibility and maintenance costs associated with the technique. For instance, the study shows that maintenance costs of frequent maintenance is lower than infrequent maintenance, positive return of investment can be achieved in reasonable time, the technique can feasibly be used over months at a time, etc (Alégroth et al 2016a). In this previous study, quantitative results from maintenance of 22 industrial grade VGT test cases are acquired, from an industrial case study, which provide detailed insights into the unique challenges associated with maintenance of VGT script logic and images over time.…”

Section: Related Workmentioning

confidence: 99%

“…Hence, challenges that are specific to Sikuli and not other VGT tools, e.g. EggPlant (TestPlant 2013), but that could be mitigated by using either VM's or remote test execution as presented in previous work (Alégroth et al 2013a;Alégroth et al 2016a).…”

Section: "The Test Data We Have Includes a Lot Of Songs Albums And Amentioning

confidence: 99%

“…Frequent maintenance also helps lower test maintenance costs since it avoids, to a larger extent, simultaneous maintenance of both logic and images that is more complex than logic or images separately (Alégroth et al 2016a). As such, a maintenance process should be integrated into the company's overall development process to ensure that changes to the SUT have not caused scripts to break.…”

Section: Long-term Use Of Vgt In Practicementioning

confidence: 99%

See 2 more Smart Citations

On the long-term use of visual gui testing in industrial practice: a case study

Alégroth

Feldt

2017

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

Visual GUI Testing (VGT) is a tool-driven technique for automated GUI-based testing that uses image recognition to interact with and assert the correctness of the behavior of a system through its GUI as it is shown to the user. The technique's applicability, e.g. defect-finding ability, and feasibility, e.g. time to positive return on investment, have been shown through empirical studies in industrial practice. However, there is a lack of studies that evaluate the usefulness and challenges associated with VGT when used long-term (years) in industrial practice. This paper evaluates how VGT was adopted, applied and why it was abandoned at the music streaming application development company, Spotify, after several years of use. A qualitative study with two workshops and five well chosen employees is performed at the company, supported by a survey, which is analyzed with a grounded theory approach to answer the study's three research questions. The interviews provide insights into the challenges, problems and limitations, but also benefits, that Spotify experienced during the adoption and use of VGT. However, due to the technique's drawbacks, VGT has been abandoned for a new technique/framework, simply called the Test interface. The Test interface is considered more robust and flexible for Spotify's needs but has several drawbacks, including that it does not test the actual GUI as shown to the user like VGT does. From the study's results it is concluded that VGT can be used long-term in industrial practice but it requires organizational change as well as engineering best practices to be beneficial. Through synthesis of the study's results, and results from previous work, a set of guidelines are presented that aim to aid practitioners to adopt and use VGT in industrial practice. However, due to the abandonment of the technique, future research is required to analyze in what types of projects the technique is, and is not, long-term viable. To this end, we also present Spotify's Test interface solution for automated GUI-based testing and conclude that it has its own benefits and drawbacks.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: "The Test Data We Have Includes a Lot Of Songs Albums And Amentioning

confidence: 99%

Section: Long-term Use Of Vgt In Practicementioning

confidence: 99%

See 1 more Smart Citation

On the long-term use of visual gui testing in industrial practice: a case study

Alégroth

Feldt

2017

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

show abstract

“…A few companies mentioned creating new tests instead of updating the existing ones in case the change is non-trivial. This could, however, be costlier than updating the old scripts as shown by [21] [22].…”

Section: E Agile Development Not Only Changed the Format Of Requiremmentioning

confidence: 99%

An Exploratory Study on Handling Requirements and Acceptance Test Documentation in Industry

Hotomski

Charrada

Glinz

2016

2016 IEEE 24th International Requirements Engineering Conference (RE)

View full text Add to dashboard Cite

With the emergence and spread of agile processes, the practices of writing and maintaining documentation have drastically changed in the last decade. In this work, we performed a qualitative study to explore the current practices for managing two related types of software documentation: requirements and acceptance tests. We interviewed twenty practitioners from seventeen business units in fifteen companies to investigate the companies' practices for writing, maintaining and linking requirements and acceptance test documentation. The study yields interesting and partially unexpected results. For example, we had expected that tests would be more extensively documented than requirements, while we found a strong linear correlation between the number of requirements and tests in our sample. We also found that technical people are usually not involved in the requirements engineering activities, which often results in misunderstood or underestimated requirements. Acceptance tests are written, in many cases, based on requirements that are not necessarily detailed enough. Also, acceptance tests are not regularly maintained, which occasionally results in confusing features and bugs. Abstract-With the emergence and spread of agile processes, the practices of writing and maintaining documentation have drastically changed in the last decade. In this work, we performed a qualitative study to explore the current practices for managing two related types of software documentation: requirements and acceptance tests. We interviewed twenty practitioners from seventeen business units in fifteen companies to investigate the companies' practices for writing, maintaining and linking requirements and acceptance test documentation. The study yields interesting and partially unexpected results. For example, we had expected that tests would be more extensively documented than requirements, while we found a strong linear correlation between the number of requirements and tests in our sample. We also found that technical people are usually not involved in the requirements engineering activities, which often results in misunderstood or underestimated requirements. Acceptance tests are written, in many cases, based on requirements that are not necessarily detailed enough. Also, acceptance tests are not regularly maintained, which occasionally results in confusing features and bugs.

show abstract

“…In this case, we set very broad priors (generic weakly informative priors) according to recommended best practices. 8 Note that, because versions are a mixture of days and weeks in this case study (see Fig. 3), an estimation error is introduced into the model However, this does not affect the estimation of the required number of hours until ROI, which can directly be derived from the y-axis of the plot.…”

Section: Discussion and Lessons Learnedmentioning

confidence: 99%

Estimating Return on Investment for GUI Test Automation Frameworks

Dobslaw

Feldt

Michaelsson

et al. 2019

2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE)

Self Cite

View full text Add to dashboard Cite

Automated graphical user interface (GUI) tests can reduce manual testing activities and increase test frequency. This motivates the conversion of manual test cases into automated GUI tests. However, it is not clear whether such automation is cost-effective given that GUI automation scripts add to the code base and demand maintenance as a system evolves. In this paper, we introduce a method for estimating maintenance cost and Return on Investment (ROI) for Automated GUI Testing (AGT). The method utilizes the existing source code change history and can be used for evaluation also of other testing or quality assurance automation technologies. We evaluate the method for a real-world, industrial software system and compare two fundamentally different AGT tools, namely Selenium and EyeAutomate, to estimate and compare their ROI. We also report on their defect-finding capabilities and usability. The quantitative data is complemented by interviews with employees at the case company. The method was successfully applied and estimated maintenance cost and ROI for both tools are reported. Overall, the study supports earlier results showing that implementation time is the leading cost for introducing AGT. The findings further suggest that while EyeAutomate tests are significantly faster to implement, Selenium tests require more of a programming background but less maintenance.

show abstract

Maintenance of automated test suites in industry: An empirical study on Visual GUI Testing

Cited by 50 publications

References 32 publications

On the long-term use of visual gui testing in industrial practice: a case study

On the long-term use of visual gui testing in industrial practice: a case study

An Exploratory Study on Handling Requirements and Acceptance Test Documentation in Industry

Estimating Return on Investment for GUI Test Automation Frameworks

Contact Info

Product

Resources

About