IconIntent: Automatic Identification of Sensitive UI Widgets Based on Icon Classification for Android Apps

Xiao, Xusheng; Wang, Xiaoyin; Zhihao, Cao; Wang, Hanlin; Gao, Peng

doi:10.1109/icse.2019.00041

Cited by 55 publications

(25 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lastly, we infer semantic information from UI elements to improve the screen reader experience. To recognize icons, previous work [50,76,77] trained image classication models from UI design datasets [27]. To describe content in pictures, prior work used deep learning models to generate natural language descriptions of images [44,46], and some accessibility improvement research has also leveraged crowdsourcing to generate image captions [35,39,40].…”

Section: Understanding Ui Semanticsmentioning

confidence: 99%

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

Zhang

Greef

Swearngin

et al. 2021

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sucient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reect an app's full functionality. We trained a robust, fast, memory-ecient, on-device model to detect UI elements using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected and annotated. To further improve UI detections and add semantic information, we introduced heuristics (e.g., UI grouping and ordering) and additional models (e.g., recognize UI content, state, interactivity). We built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver. In a study with 9 screen reader users, we validated that our approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used. CCS CONCEPTS• Human-centered computing ! Accessibility technologies.

show abstract

Section: Understanding Ui Semanticsmentioning

confidence: 99%

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

Zhang

Greef

Swearngin

et al. 2021

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…The deep neural network is modified from the research of Xiao et al [5]. The model encodes the widget screenshot into a feature vector with a convolutional neural network (CNN).…”

Section: Report #25mentioning

confidence: 99%

Prioritize Crowdsourced Test Reports via Deep Screenshot Understanding

Fang

Cao

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

Crowdsourced testing is increasingly dominant in mobile application (app) testing, but it is a great burden for app developers to inspect the incredible number of test reports. Many researches have been proposed to deal with test reports based only on texts or additionally simple image features. However, in mobile app testing, texts contained in test reports are condensed and the information is inadequate. Many screenshots are included as complements that contain much richer information beyond texts. This trend motivates us to prioritize crowdsourced test reports based on a deep screenshot understanding.In this paper, we present a novel crowdsourced test report prioritization approach, namely D e e p P r io r . We first represent the crowdsourced test reports with a novelly introduced feature, namely D e e p F e a t u r e , that includes all the widgets along with their texts, coordinates, types, and even intents based on the deep analysis of the app screenshots, and the textual descriptions in the crowdsourced test reports. D e e p F e a t u r e includes the Bug Feature, which directly describes the bugs, and the Context Feature, which depicts the thorough context of the bug. The similarity of the D e e p F e a t u r e is used to represent the test reports' similarity and prioritize the crowdsourced test reports. We formally define the similarity as D e e p Si m i l a r i t y . We also conduct an empirical experiment to evaluate the effectiveness of the proposed technique with a large dataset group. The results show that D e e p P r io r is promising, and it outperforms the stateof-the-art approach with less than half the overhead.

show abstract

“…Articles presenting only data sets have been excluded [Pandian et al 2020a][Deka et al 2017. We also excluded articles that focus exclusively on the detection of specific GUI elements, such as icons [Xiao et al 2019]. We excluded research directed rather on the retrieval of GUIs [Chen et al 2019b, generation of GUI design patterns [Nguyen et al 2018], or semantic annotations [Liu et al 2018a] rather than code.…”

Section: Quantitymentioning

confidence: 99%

Recent Progress in Automated Code Generation from GUI Images Using Machine Learning Techniques

Baulé¹,

Wangenheim²,

Wangenheim³

et al. 2020

jucs

View full text Add to dashboard Cite

The manual transformation of a user interface design into code is a costly and time-consuming process. A solution can be the automation of the generation of code based on sketches or GUI design images. Recently, Machine Learning approaches have shown promising results in detecting GUI elements for such automation. Thus, to provide an overview of existing approaches, we performed a systematic mapping study. As a result, we identified and compared 20 approaches, that demonstrate good performance results being considered useful. These results can be used by researchers and practitioners in order to improve the efficiency of the GUI design process as well as continue to evolve and improve approaches for its support.

show abstract

IconIntent: Automatic Identification of Sensitive UI Widgets Based on Icon Classification for Android Apps

Cited by 55 publications

References 30 publications

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

Prioritize Crowdsourced Test Reports via Deep Screenshot Understanding

Recent Progress in Automated Code Generation from GUI Images Using Machine Learning Techniques

Contact Info

Product

Resources

About