Transitioning Manual System Test Suites to Automated Testing: An Industrial Case Study

Alégroth, Emil; Feldt, Robert; Olsson, Helena Holmström

doi:10.1109/icst.2013.14

Cited by 25 publications

(48 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…test synchronization to the SUT [Horowitz and Singhera(1993)] and usage of faulty manual test specifications that lead to faulty test results regardless of how they are executed. This result is a significant since other GUI-based testing techniques are used successfully in industrial practice, which would corroborate our previous research results that VGT is industrially applicable [Börjesson and Feldt(2012), Alegroth et al(2013a) Alegroth, Feldt, and Olsson]. However, other GUI-based techniques have been reported to be associated with high maintenance costs [Sjösten- Andersson and Pareto(2006)], costs that are still unknown for VGT.…”

Section: Test Tool Deficiencies (20-23)supporting

confidence: 79%

“…10 percent of the tested system's manual test cases were automated with both tools, which showed that there was no statistical significant difference between the tools and that both tools were fully capable of performing the automation with equal defect finding ability as the manual test cases. However, the study was performed by academic experts in VGT and therefore a second study was performed, driven by industrial practitioners in an industrial project [Alegroth et al(2013a) Alegroth, Feldt, and Olsson]. Results of the second study showed that VGT is also applicable when performed in a real project environment when used by industrial practitioners under real-world time and cost constraints.…”

Section: Background and Related Workmentioning

confidence: 99%

“…Maintenance costs that are currently unknown. Only initial data has been acquired [Alegroth et al(2013a) Alegroth, Feldt, and Olsson], and more studies that focus on these costs are therefore required in the future.…”

Section: -6mentioning

confidence: 99%

“…After the evaluation, Sikuli [Yeh et al(2009)Yeh, Chang, and Miller] was identified as the most suitable alternative for the transition. Further information regarding the VGT tool selection can be found in [Alegroth et al(2013a) Alegroth, Feldt, and Olsson].…”

Section: Detailed Data Collection In Casementioning

confidence: 99%

“…First, to evaluate if VGT could be applied in practice in an industrial project, results presented in [Alegroth et al(2013a) Alegroth, Feldt, and Olsson]. Second, to evaluate the CPLs experienced by industrial practitioners during VGT transition in this context.…”

Section: Industrial Case Studymentioning

confidence: 99%

See 4 more Smart Citations

Visual GUI testing in practice: challenges, problemsand limitations

2014

Self Cite

View full text Add to dashboard Cite

In today's software development industry, high-level tests such as Graphical User Interface (GUI) based system and acceptance tests are mostly performed with manual practices that are often costly, tedious and error prone. Test automation has been proposed to solve these problems but most automation techniques approach testing from a lower level of system abstraction. Their suitability for high-level tests has therefore been questioned. High-level test automation techniques such as Record and Replay exist, but studies suggest that these techniques suffer from limitations, e.g. sensitivity to GUI layout or code changes, system implementation dependencies, etc.Visual GUI Testing (VGT) is an emerging technique in industrial practice with perceived higher flexibility and robustness to certain GUI changes than previous high-level (GUI) test automation techniques. The core of VGT is image recognition which is applied to analyze and interact with the bitmap layer of a system's front end. By coupling image recognition with test scripts, VGT tools can emulate end user behavior on almost any GUI-based system, regardless of implementation language, operating system or platform. However, VGT is not without its own challenges, problems and limitations (CPLs) but, like for many other automated test techniques, there is a lack of empirically-based knowledge of these CPLs and how they impact industrial applicability. Crucially, there is also a lack of information on the cost of applying this type of test automation in industry.This manuscript reports an empirical, multi-unit case study performed at two Swedish companies that develop safety-critical software. It studies their transition from manual system test cases into tests automated with VGT. In total, four different test suites that together include more than 300 high-level system test cases were automated for two multi-million lines of code systems. The results show that the transitioned test cases could find defects in the tested systems and that all applicable test cases could be automated. However, during these transition projects a number of hurdles had to be addressed; a total of 58 different CPLs were identified and then categorized into 26 types. We present these CPL types and an analysis of the implications for the transition to and use of VGT in industrial software development practice. In addition, four high-level solutions are presented that were identified during the study, which would address about half of the identified CPLs.Furthermore, collected metrics on cost and return on investment of the VGT transition are reported together with information about the VGT suites' defect finding ability. Nine of the identified defects are reported, 5 of which were unknown to testers with extensive experience from using the manual test suites. The main conclusion from this study is that even though there are many challenges related to the transition and usage of VGT, the technique is still valuable, flexible and considered cost-effective by the industrial practit...

show abstract

Section: Test Tool Deficiencies (20-23)supporting

confidence: 79%

Section: Background and Related Workmentioning

confidence: 99%

Section: -6mentioning

confidence: 99%

Section: Detailed Data Collection In Casementioning

confidence: 99%

Section: Industrial Case Studymentioning

confidence: 99%

See 3 more Smart Citations

Visual GUI testing in practice: challenges, problemsand limitations

2014

Self Cite

View full text Add to dashboard Cite

show abstract

Impediments for software test automation: A systematic literature review

Wiklund

Eldh

Sundmark

et al. 2017

Software Testing Verif & Rel

View full text Add to dashboard Cite

SummaryAutomated software testing is a critical enabler for modern software development, where rapid feedback on the product quality is expected. To make the testing work well, it is of high importance that impediments related to test automation are prevented and removed quickly. An enabling factor for all types of improvement is to understand the nature of what is to be improved. We have performed a systematic literature review of reported impediments related to software test automation to contribute to this understanding. In this paper, we present the results from the systematic literature review: The list of identified publications, a categorization of identified impediments, and a qualitative discussion of the impediments proposing a socio-technical system model of the use and implementation of test automation. 30% to 80% of the development costs are reported to be related to testing [2,3] . The high use of test automation also makes the automated tests mission critical: If the test automation stops working or slows down, the development will also stop or slow down.The investment cost for test automation can be significant. It is difficult to propose a generalizable cost model, due to a lack of published information available on the cost of ownership for test automation. However, the size and complexity of a test system can be in the order of, or often larger than, the complexity and size of the tested product [10,11][IP7] [12]. This makes it reasonable to conjecture that the cost of ownership of an automated test execution system is in the order of the cost of development of the product it is testing. Even if a lot less effort is spent on the test systems than on the sold products, it still represents a major investment for the software business as a whole. If the test automation does not perform as well as expected, this investment is unlikely to be recovered as planned, which may have significant consequences for the business. Further, it is reasonable to conjecture that test systems suffer from similar technical debt issues as regular systems do. Technical debt is a term that is used to describe the gap between the current state of the system and the ideal state of the system [13], usually in situations where a compromise is made during development to meet demands in one dimension, such as lead time, by sacrificing work in another dimension, such as architecture.

show abstract

Improving Web Element Localization by Using a Large Language Model

Nass,

Alégroth,

Feldt

2024

Software Testing Verif & Rel

View full text Add to dashboard Cite

Web‐based test automation heavily relies on accurately finding web elements. Traditional methods compare attributes but do not grasp the context and meaning of elements and words. The emergence of large language models (LLMs) like GPT‐4, which can show human‐like reasoning abilities on some tasks, offers new opportunities for software engineering and web element localization. This paper introduces and evaluates VON Similo LLM, an enhanced web element localization approach. Using an LLM, it selects the most likely web element from the top‐ranked ones identified by the existing VON Similo method, ideally aiming to get closer to human‐like selection accuracy. An experimental study was conducted using 804 web element pairs from 48 real‐world web applications. We measured the number of correctly identified elements as well as the execution times, comparing the effectiveness and efficiency of VON Similo LLM against the baseline algorithm. In addition, motivations from the LLM were recorded and analysed for 140 instances. VON Similo LLM demonstrated improved performance, reducing failed localizations from 70 to 40 (out of 804), a 43% reduction. Despite its slower execution time and additional costs of using the GPT‐4 model, the LLM's human‐like reasoning showed promise in enhancing web element localization. LLM technology can enhance web element localization in GUI test automation, reducing false positives and potentially lowering maintenance costs. However, further research is necessary to fully understand LLMs' capabilities, limitations and practical use in GUI testing.

show abstract

Transitioning Manual System Test Suites to Automated Testing: An Industrial Case Study

Cited by 25 publications

References 13 publications

Visual GUI testing in practice: challenges, problemsand limitations

Visual GUI testing in practice: challenges, problemsand limitations

Impediments for software test automation: A systematic literature review

Improving Web Element Localization by Using a Large Language Model

Contact Info

Product

Resources

About