A Large-Scale Longitudinal Analysis of Missing Label Accessibility Failures in Android Apps

Fok, Raymond; Zhong, Mingyuan; Ross, Anne Spencer; Fogarty, James; Wobbrock, Jacob O.

doi:10.1145/3491102.3502143

Cited by 12 publications

(2 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We design a novel tool that automatically navigates through these apps, mining a large number of UIs in the process. Recent extensive studies [36,39] reveal that even the most recent apps can still harbor noise and errors. To ensure the dataset's quality, we utilize mature techniques to carefully filter out noise and errors in advance.…”

Section: Datasets For Ui Modelingmentioning

confidence: 99%

MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling

Feng,

Ma,

Wang

et al. 2024

Proceedings of the CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

The importance of computational modeling of mobile user interfaces (UIs) is undeniable. However, these require a high-quality UI dataset. Existing datasets are often outdated, collected years ago, and are frequently noisy with mismatches in their visual representation. This presents challenges in modeling UI understanding in the wild. This paper introduces a novel approach to automatically mine UI data from Android apps, leveraging Large Language Models (LLMs) to mimic human-like exploration. To ensure dataset quality, we employ the best practices in UI noise filtering and incorporate human annotation as a final validation step. Our results demonstrate the effectiveness of LLMs-enhanced app exploration in mining more meaningful UIs, resulting in a large dataset MUD of 18k human-annotated UIs from 3.3k apps. We highlight the usefulness of MUD in two common UI modeling tasks: element detection and UI retrieval, showcasing its potential to establish a foundation for future research into high-quality, modern UIs. CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI).

show abstract

Section: Datasets For Ui Modelingmentioning

confidence: 99%

MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling

Feng,

Ma,

Wang

et al. 2024

Proceedings of the CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…Our highly automated data collection process also allows WebUI to be more easily updated in the future by re-visiting the same list of URLs. An updated version of the dataset could also facilitate longitudinal analysis of the design [14] and accessibility [21] of web UIs. Nevertheless, WebUI is currently unlikely to support other types of modeling, such as user interaction mining [15,16], that require realistic interaction traces, since our crawling strategy was largely based on random link traversal.…”

Section: Performance Impact Of Web Datamentioning

confidence: 99%

WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics

Wu¹,

Wang²,

Shen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Modeling user interfaces (UIs) from visual information allows systems to make inferences about the functionality and semantics needed to support use cases in accessibility, app automation, and testing. Current datasets for training machine learning models are limited in size due to the costly and time-consuming process of manually collecting and annotating UIs. We crawled the web to construct WebUI, a large dataset of 400,000 rendered web pages associated with automatically extracted metadata. We analyze the composition of WebUI and show that while automatically extracted data is noisy, most examples meet basic criteria for visual UI modeling. We applied several strategies for incorporating semantics found in web pages to increase the performance of visual UI understanding models in the mobile domain, where less labeled data is available: (i) element detection, (ii) screen classification and (iii) screen similarity.

show abstract

Slide Gestalt: Automatic Structure Extraction in Slide Decks for Non-Visual Access

Peng

Chi

Kannan

et al. 2023

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

A Large-Scale Longitudinal Analysis of Missing Label Accessibility Failures in Android Apps

Cited by 12 publications

References 30 publications

MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling

MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling

WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics

Slide Gestalt: Automatic Structure Extraction in Slide Decks for Non-Visual Access

Contact Info

Product

Resources

About