JAutomate: A Tool for System- and Acceptance-test Automation

Alégroth, Emil; Nass, Michel; Olsson, Helena Holmström

doi:10.1109/icst.2013.61

Cited by 39 publications

(39 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sikuli [Chang et al(2010)Chang, Yeh, and Miller], and commercial, e.g. JAutomate [Alegroth et al(2013b) Alegroth, Nass, and Olsson], but still the technique is only sparsely used in industry. VGT is performed by either manual development, or recording, of scripts that define the intended interaction with the SUT, usually defined as scenarios, which also include images of the bitmap-components the tools should interact with, e.g.…”

Section: Background and Related Workmentioning

confidence: 99%

“…The reason is because most VGT tools, e.g. JAutomate [Alegroth et al(2013b)Alegroth, Nass, and Olsson] and EggPlant [TestPlant(2013)], have built in support for test suite creation and maintenance. As such, these tools also have built in failure mitigation and exception handling support.…”

Section: The Vgt Suitesmentioning

confidence: 99%

“…However, other tools, e.g. JAutomate [Alegroth et al(2013b) Alegroth, Nass, and Olsson] and EggPlant [TestPlant(2013)], also have built in failure mitigation support such as multiple image recognition algorithms, etc. Failure mitigation that relieves some, but not all, of the need for custom robustness improvements and exception handling solutions.…”

Section: Potential Cpl Solutionsmentioning

confidence: 99%

“…JAutomate [Alegroth et al(2013b) Alegroth, Nass, and Olsson]. However, based on the commonalities between VGT tools [Alegroth et al(2013b) Alegroth, Nass, and Olsson], this is considered a minor risk.…”

Section: Similarities Between Vgt and Other Gui-based Techniquesmentioning

confidence: 99%

“…Our list of CPLs can be used to anticipate problems and costs and help create plans for the transitions as well as for finding solutions in a new company and automation context. This contribution is of particular interest since it has been established that there is an industrial need and want for VGT [Alegroth et al(2013b) Alegroth, Nass, and Olsson]. In addition, the CPLs can help guide future research both in developing solutions and work-arounds but also in further development of testing techniques and tools that overcome them.…”

Section: Challenges Problems and Limitationsmentioning

confidence: 99%

See 4 more Smart Citations

Visual GUI testing in practice: challenges, problemsand limitations

2014

View full text Add to dashboard Cite

In today's software development industry, high-level tests such as Graphical User Interface (GUI) based system and acceptance tests are mostly performed with manual practices that are often costly, tedious and error prone. Test automation has been proposed to solve these problems but most automation techniques approach testing from a lower level of system abstraction. Their suitability for high-level tests has therefore been questioned. High-level test automation techniques such as Record and Replay exist, but studies suggest that these techniques suffer from limitations, e.g. sensitivity to GUI layout or code changes, system implementation dependencies, etc.Visual GUI Testing (VGT) is an emerging technique in industrial practice with perceived higher flexibility and robustness to certain GUI changes than previous high-level (GUI) test automation techniques. The core of VGT is image recognition which is applied to analyze and interact with the bitmap layer of a system's front end. By coupling image recognition with test scripts, VGT tools can emulate end user behavior on almost any GUI-based system, regardless of implementation language, operating system or platform. However, VGT is not without its own challenges, problems and limitations (CPLs) but, like for many other automated test techniques, there is a lack of empirically-based knowledge of these CPLs and how they impact industrial applicability. Crucially, there is also a lack of information on the cost of applying this type of test automation in industry.This manuscript reports an empirical, multi-unit case study performed at two Swedish companies that develop safety-critical software. It studies their transition from manual system test cases into tests automated with VGT. In total, four different test suites that together include more than 300 high-level system test cases were automated for two multi-million lines of code systems. The results show that the transitioned test cases could find defects in the tested systems and that all applicable test cases could be automated. However, during these transition projects a number of hurdles had to be addressed; a total of 58 different CPLs were identified and then categorized into 26 types. We present these CPL types and an analysis of the implications for the transition to and use of VGT in industrial software development practice. In addition, four high-level solutions are presented that were identified during the study, which would address about half of the identified CPLs.Furthermore, collected metrics on cost and return on investment of the VGT transition are reported together with information about the VGT suites' defect finding ability. Nine of the identified defects are reported, 5 of which were unknown to testers with extensive experience from using the manual test suites. The main conclusion from this study is that even though there are many challenges related to the transition and usage of VGT, the technique is still valuable, flexible and considered cost-effective by the industrial practit...

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Section: The Vgt Suitesmentioning

confidence: 99%

Section: Potential Cpl Solutionsmentioning

confidence: 99%

Section: Similarities Between Vgt and Other Gui-based Techniquesmentioning

confidence: 99%

Section: Challenges Problems and Limitationsmentioning

confidence: 99%

See 3 more Smart Citations

Visual GUI testing in practice: challenges, problemsand limitations

2014

View full text Add to dashboard Cite

show abstract

Pesto: Automated migration of DOM‐based Web tests towards the visual approach

Leotta

Stocco

Ricca

et al. 2018

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary Test automation tools are widely adopted for testing complex Web applications. Three generations of tools exist: first, based on screen coordinates; second, based on DOM–based commands; and third, based on visual image recognition. In our previous work, we proposed Pesto, a tool able to migrate second‐generation Selenium WebDriver test suites towards third‐generation Sikuli ones. In this work, we extend Pesto to manage Web elements having (1) complex visual interactions and (2) multiple visual appearances. Pesto relies on aspect‐oriented programming, computer vision, and code transformations. Our new improved tool has been evaluated on two Web test suites developed by an independent tester. Experimental results show that Pesto manages and transforms correctly test suites with Web elements having complex visual interactions and multistate elements. By using Pesto, the migration of existing DOM–based test suites to the visual approach requires a low manual effort, since our approach proved to be very accurate.

show abstract

Sidereal: Statistical adaptive generation of robust locators for web testing

Leotta

Ricca

Tonella

2021

Software Testing Verif & Rel

View full text Add to dashboard Cite

SummaryBy ensuring adequate functional coverage, End‐to‐End (E2E) testing is a key enabling factor of continuous integration. This is even more true for web applications, where automated E2E testing is the only way to exercise the full stack used to create a modern application. The test code used for web testing usually relies on DOM locators, often expressed as XPath expressions, to identify the web elements and to extract the data checked in assertions. When applications evolve, the most dominant cost for the evolution of test code is due to broken locators, which fail to locate the target element in the novel versions and must be repaired. In this paper, we formulate the robust XPath locator generation problem as a graph exploration problem, instead of relying on ad‐hoc heuristics as the one implemented by the state of the art tool robula+. Our approach is based on a statistical adaptive algorithm implemented by the tool sidereal, which outperforms robula+'s heuristics in terms of robustness by learning the potential fragility of HTML properties from previous versions of the application under test. sidereal was applied to six applications and to a total of 611 locators and was compared against two baseline algorithms, robula+ and Montoto. The adoption of sidereal results in a significant reduction of the number of broken locators (respectively ‐55% and ‐70%). The time for generating such robust locators was deemed acceptable being in the order of hundredths of second.

show abstract

JAutomate: A Tool for System- and Acceptance-test Automation

Cited by 39 publications

References 10 publications

Visual GUI testing in practice: challenges, problemsand limitations

Visual GUI testing in practice: challenges, problemsand limitations

Pesto: Automated migration of DOM‐based Web tests towards the visual approach

Sidereal: Statistical adaptive generation of robust locators for web testing

Contact Info

Product

Resources

About