Visual Identification of Mobile App GUI Elements for Automated Robotic Testing

Abstract

Automated robotic testing is an emerging testing approach for mobile apps that can afford complete black-box testing. Compared with other automated testing approaches, automatic robotic testing can reduce the dependence on the internal information of apps. However, capturing GUI element information accurately and effectively from a black-box perspective is a critical issue in robotic testing. This study introduces object detection technology to achieve the visual identification of mobile app GUI elements. First, we consider the requirements of test implementation, the feasibility of visual identification, and the external image features of GUI comprehensively to complete the reasonable classification of GUI elements. Subsequently, we constructed and optimized an object detection dataset for the mobile app GUI. Finally, we implement the identification of GUI elements based on the YOLOv3 model and evaluate the effectiveness of the results. This work can serve as the basis for vision-driven robotic testing for mobile apps and presents a universal approach that is not restricted by platforms to identify mobile app GUI elements.

Apps as partial replacement for robotics and automation systems in construction health and safety management

“…Recent research (Yap et al ., 2021) has suggested that robotics and automation can reduce construction accidents. These technologies improve hazard detection, safety planning and inspections and increase safety awareness (Xue et al ., 2022; Yap et al ., 2021; Hou et al ., 2020). Wearable technologies can also monitor workers' health and detect early signs of overwork or stress (Xue et al ., 2022).…”

“…These technologies improve hazard detection, safety planning and inspections and increase safety awareness (Xue et al ., 2022; Yap et al ., 2021; Hou et al ., 2020). Wearable technologies can also monitor workers' health and detect early signs of overwork or stress (Xue et al ., 2022). However, construction adoption faces challenges, especially cost issues, especially for small- and medium-sized businesses, mainly in developing countries (Salento, 2017; Karakhan et al ., 2018; Xue et al ., 2022; Wu et al ., 2021, 2023; Chauhan, 2021; Okpala et al ., 2021).…”

“…Wearable technologies can also monitor workers' health and detect early signs of overwork or stress (Xue et al ., 2022). However, construction adoption faces challenges, especially cost issues, especially for small- and medium-sized businesses, mainly in developing countries (Salento, 2017; Karakhan et al ., 2018; Xue et al ., 2022; Wu et al ., 2021, 2023; Chauhan, 2021; Okpala et al ., 2021). Financial hurdles hinder the implementation of robotics and automation for health and safety improvements.…”

Retracted: Visual Identification of Mobile App GUI Elements for Automated Robotic Testing

“…Tis article has been retracted by Hindawi following an investigation undertaken by the publisher [1]. Tis investigation has uncovered evidence of one or more of the following indicators of systematic manipulation of the publication process:…”

“…This article has been retracted by Hindawi following an investigation undertaken by the publisher [ 1 ]. This investigation has uncovered evidence of one or more of the following indicators of systematic manipulation of the publication process: Discrepancies in scope Discrepancies in the description of the research reported Discrepancies between the availability of data and the research described Inappropriate citations Incoherent, meaningless and/or irrelevant content included in the article Peer-review manipulation …”