GUI testing using computer vision

Chang, Tsung-Hsiang; Yeh, Tom; Miller, Robert C.

doi:10.1145/1753326.1753555

Cited by 166 publications

(90 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The authors demonstrate a variety of tasks programmed using Sikuli, ranging from the automation of repetitious tasks, to macros for closing pop-ups. The authors also use Sikuli's programming tools to implement a general-purpose contextual help enhancement [75] and a sophisticated UI testing framework [12]. Sikuli uses computer vision methods (template matching and voting based on invariant local features), which are some of the most advanced pixel-based methods used today.…”

Section: General-purpose Methodsmentioning

confidence: 99%

Pixel-based reverse engineering of graphical interfaces

Dixon

2013

Proceedings of the Adjunct Publication of the 26th Annual ACM Symposium on User Interface Software and Technology - UIST '13 Ad

View full text Add to dashboard Cite

Pixel-Based Reverse Engineering of Graphical Interfaces Morgan Emory DixonChair of the Supervisory Committee: James Fogarty Computer Science & EngineeringUser interface software is often large, complex, rigid, and difficult to implement. In fact, most of the development time and effort in a typical application is dedicated to the user interface. Not only do interfaces require the programmer to deal with complex input and output, asynchrony, and rapid feedback to the user that is context-dependent, but they also need to run on many different devices and serve a wide variety of people with complex needs. For example, applications may be required to run on the desktop, on mobile touchscreen devices, both for people with motor impairments and able-bodied users, and in multiple languages. As a result, most of today's interfaces require an immense amount of resources to address the increasingly complex ecosystem of devices and users.This dissertation focuses on unlocking existing user interfaces to overcome these challenges. Similar to how Wikipedia opened encyclopedia articles for anyone to edit, we envision a similar democratization of user interfaces, where anybody can modify the graphical interface of any application. Any single company or group of developers cannot single-handedly address people's wide variety of needs, but unlocking existing interfaces could diffuse the cost and effort across many people. As a first step towards realizing this vision, this dissertation explores how to modify existing graphical interfaces by reverse engineering their pixels. Working from the pixels enables developers and end-users to modify existing interfaces without compliance from the underlying application. For example, we use our pixel-based methods to implement many different modifications including accessibility enhancements, improved input on mobile devices, interface translation for improved localization, and end-user customization.Researchers have explored pixel-based reverse engineering for decades, but this dissertation presents the first real-time methods for interpreting an interface's pixels and using that reverse engineered structure to modify an application's apparent behavior.

show abstract

Section: General-purpose Methodsmentioning

confidence: 99%

Pixel-based reverse engineering of graphical interfaces

Dixon

2013

Proceedings of the Adjunct Publication of the 26th Annual ACM Symposium on User Interface Software and Technology - UIST '13 Ad

View full text Add to dashboard Cite

show abstract

“…A group of web testing techniques (e.g., Cucumber [11], Sikuli [6,61], Crawljax [33], Selenium [46], Cornipickle [16]) require developers to manually write test cases or specify invariants to be checked against the application. However, unless developers exhaustively specify a correctness variant for each element and style combination, they cannot be reliably used to localize faults and x them.…”

Section: Related Workmentioning

confidence: 99%

Automated repair of layout cross browser issues using search-based techniques

Mahajan

Alameer

McMinn

et al. 2017

Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

A consistent cross-browser user experience is crucial for the success of a website. Layout Cross Browser Issues (XBIs) can severely undermine a website's success by causing web pages to render incorrectly in certain browsers, thereby negatively impacting users' impression of the quality and services that the web page delivers. Existing Cross Browser Testing (XBT) techniques can only detect XBIs in websites. Repairing them is, hitherto, a manual task that is labor intensive and requires signi cant expertise. Addressing this concern, our paper proposes a technique for automatically repairing layout XBIs in websites using guided search-based techniques. Our empirical evaluation showed that our approach was able to successfully x 86% of layout XBIs reported for 15 di erent web pages studied, thereby improving their cross-browser consistency. CCS CONCEPTS•So ware and its engineering →So ware testing and debugging; Search-based so ware engineering; KEYWORDSCross-browser issues; automated search-based repair; web apps. ACM Reference format:

show abstract

“…In the latter case, where it is not possible to have access to application's source code, there is a need for an alternative way to carry out automation tasks. In those situations Picture-Driven Computing is seen as a viable alternative (Chang, Tsung-Hsiang et al, 2010). This new paradigm offers software developers a fully logical alternative.…”

Section: B Picture-driven Computingmentioning

confidence: 99%

Supporting GUI Exploration through USS Tool

Silva

Ornelas

Silva

2016

Journal of Information Systems Engineering & Management

View full text Add to dashboard Cite

Advances in usability and design techniques (e.g. user-centered design) try to facilitate the use of interactive systems. However, users still have to adapt to interactive systems, i.e. they have to learn the steps required to accomplish a task either by trial and error or by obtaining help. While advanced users are usually able to adapt without much effort this is far from being the case with beginners. Some interactive systems offer different interaction styles in an attempt to meet the needs of all types of user but this is not the case with all interactive systems. In this sense, we present an approach to support the use of any interactive system making use of enriched models and picture-driven computing to achieve tasks automation. The USS tool (User Support System) is the basis to the adaptation of interactive systems accordingly to the users' needs. The approach provides the foundation for the addition of help (based on demonstration) to any graphical user interfaces (GUI) facilitating learning and use. The work is illustrated by a case study and completed with a preliminary user evaluation which provides insights about the validity of the approach.

show abstract

GUI testing using computer vision

Cited by 166 publications

References 17 publications

Pixel-based reverse engineering of graphical interfaces

Pixel-based reverse engineering of graphical interfaces

Automated repair of layout cross browser issues using search-based techniques

Supporting GUI Exploration through USS Tool

Contact Info

Product

Resources

About