Why many challenges with GUI test automation (will) remain

Nass, Michel; Alégroth, Emil; Feldt, Robert

doi:10.1016/j.infsof.2021.106625

Cited by 24 publications

(12 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Automatic testing of GUI designs could be employed to detect inconsistency issues. Widget detection in GUIs has been identified as one of the biggest challenges in automatic GUI testing [ 114 ]. This type of test is important, as they allow gathering information about triggering sequences of GUI failures.…”

Section: Discussionmentioning

confidence: 99%

Automated Workflow for Usability Audits in the PHR Realm

García‐Berná

Sobrino-Duque

Gea

et al. 2022

IJERPH

View full text Add to dashboard Cite

Teleworking is an everyday reality in today’s world. Many companies are opting for this way of working remotely, as it improves employers’ satisfaction along with the work–life balance and it does not lead to any significant reduction in performance. In general, the comfortable use of software applications is a priority, and quality aspects such as usability are of vital importance. In this paper, a novel workflow is proposed. It consists of the following components: (1) a tool to manage usability evaluations based on expert reviews, called Usevalia; (2) a set of heuristics to be considered when performing the evaluations; (3) a catalogue of usability requirements to guide the evaluations; (4) a checklist related to the heuristics to verify that the evaluations have been performed correctly by all the auditors; and (5) a set of tasks to know in advance the functionality of the applications to be audited. Personal health records (PHRs) were taken as a case study. This methodology made it possible to involve a total of 31 working groups that evaluated the usability of four PHRs. Concerning the main results, the evaluators agreed that with Usevalia the usability evaluations could go a step further because they are coordinated and can work together remotely. In addition, analyses of usability scores provided insight into current proposals for improving usability of applications.

show abstract

Section: Discussionmentioning

confidence: 99%

Automated Workflow for Usability Audits in the PHR Realm

García‐Berná

Sobrino-Duque

Gea

et al. 2022

IJERPH

View full text Add to dashboard Cite

show abstract

“…Pure image pixel comparison is ineffective. We expect that the robot can focus (i) Input: recorded video RV (ii) Output: test sequence script TSS (1) split_tag � 0 (2) for each frame in RV do (3) finger � getOpenPoseDetection(RV) (4) if finger is exist then (5) gesture_info ⇐ map(frame, split_tag, finger) ( 6) else (7) gui_info ⇐ map(frame, split_tag) (8) if pre_frames is gesture frame then (9) split_tag ++ (10) end if (11) end if (12) end for (13) for st � 0 to split_tag do (14) gui � getMiddleFrame(gui_info, st) (15) gui_elements � getObjectDetection(gui) (16) gui_skeleton � getGuiSkeleton(gui_elements) (17) gesture_micro � identifyGesture(gesture_info, st) (18) object � identifyObject(gesture_micro, gui_elements) (19) TTS ⇐ getTestSquence(st, gui_skeleton, gesture_micro, object) (20) end for (21) gui_elements' � getObjectDetection(gui_current) (5) gui_skeleton' � getGUISkeleton(gui_elements') (6) if serial_number is 0 then (7) break (8) else if serial_number is 1 then (9) for each tss in TSSs do (10) gui_skeleton � getTSSGUI(tss, serial_number) (11) e � getSimilarityJudgment(gui_skeleton', gui_skeleton) (12) end for (13) TSS' � getMaxSimliarScript(e, s_threshold) (14) serial_number � execteAction(TSS', serial_number, gui_elements') ( 15) else (16) gui_skeleton � getTSSGUI(TSS′, serial_number) (17) e � getSimilarityJudgment(gui_skeleton', gui_skeleton) (18) if e > s_threshold then (19) TR ⇐ recordResults() (20) serial_number � exectueAction(TSS', serial_number, gui_elements') ( 21) else (2...…”

Section: Gui Identification and Judgementmentioning

confidence: 99%

“…Differences in screen size, resolution, operating system type, and operating system version of mobile devices will lead to the failure of record and replay. In actual use, the maintenance workload for the record scripts to ensure the validity of the record-replay testing is still huge [20]. In addition, multiplatform version apps require independent record-replay testing tools for each platform, which will also increase the cost and complexity of testing.…”

Section: Introductionmentioning

confidence: 99%

Learning-Replay Based Automated Robotic Testing for Mobile App

Feng

Zhang

2022

Mobile Information Systems

View full text Add to dashboard Cite

Record-replay testing is widely used in mobile app testing as an automated testing method. However, the current record-replay methods are closely dependent on the internal information of the device or app under test. Due to the diversity of mobile devices and system platforms, their practical use is limited. To break this limitation, this paper proposes an entirely black-box learning-replay testing approach by combining robotics and vision technology to achieve a record-replay testing that can support cross-device and cross-platform. Firstly, vision technology is used to extract the critical information of GUI and gesture actions during the tester’s testing process; secondly, the GUI composition and test actions are analyzed to form a test sequence; finally, the robotic arm is guided to complete the replay of the test sequence through visual judgment. On the one hand, the approach in this paper does not access the interior of the app, shielding the association between test actions and device; on the other hand, it captures more abstract test action information instead of simple operation location records and supports more flexible test action replay. We demonstrate the effectiveness of this approach by evaluating the learning-replay of 12 popular apps for 13 typical scenarios on the same device, across devices, and across platforms.

show abstract

“…system testing through the SUT's GUI [6]. However, despite continued research since the 1980s, several key challenges remain and limit the widespread adoption of automated GUI testing [7]. One of these challenges is the robust identification of GUI elements.…”

Section: Introductionmentioning

confidence: 99%

“…Despite its importance, research has had marginal success in solving the challenge of robust GUI element localization [7]. Instead, much research has been focused on extending GUI testing technologies, only utilizing the already available web element localization solutions-example of such extensions are test generation and GUI ripping [13], [14].…”

Section: Introductionmentioning

confidence: 99%

Robust web element identification for evolving applications by considering visual overlaps

Nass¹,

Coppola²,

Alégroth³

et al. 2023

Preprint

View full text Add to dashboard Cite

Fragile (i.e., non-robust) test execution is a common challenge for automated GUI-based testing of web applications as they evolve. Despite recent progress, there is still room for improvement since test execution failures caused by technical limitations result in unnecessary maintenance costs that limit its effectiveness and efficiency. One of the most reported technical challenges for web-based tests concerns how to reliably locate a web element used by a test script.This paper proposes the novel concept of Visually Overlapping Nodes (VON) that reduces fragility by utilizing the phenomenon that visual web elements (observed by the user) are constructed from multiple web-elements in the Document Object Model (DOM) that overlaps visually.We demonstrate the approach in a tool, VON Similo, which extends the state-of-the-art multi-locator approach (Similo) that is also used as the baseline for an experiment. In the experiment, a ground truth set of 1163 manually collected web element pairs, from different releases of the 40 most popular web applications on the internet, are used to compare the approaches' precision, recall, and accuracy.Our results show that VON Similo provides 94.7% accuracy in identifying a web element in a new release of the same SUT. In comparison, Similo provides 83.8% accuracy.These results demonstrate the applicability of the visually overlapping nodes concept/tool for web element localization in evolving web applications and contribute a novel way of thinking about web element localization in future research on GUI-based testing.

show abstract

Why many challenges with GUI test automation (will) remain

Cited by 24 publications

References 24 publications

Automated Workflow for Usability Audits in the PHR Realm

Automated Workflow for Usability Audits in the PHR Realm

Learning-Replay Based Automated Robotic Testing for Mobile App

Robust web element identification for evolving applications by considering visual overlaps

Contact Info

Product

Resources

About