Erica

Deka, Biplab; Huang, Zifeng; Kumar, Ranjitha

doi:10.1145/2984511.2984581

Cited by 71 publications

(14 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To identify tappability issues automatically, we needed to collect data on a large scale to allow us to use a machine learning approach for this problem. Recently, data-driven approaches have been used to identify usability issues [8], and collect mobile app design data at scale [7,9]. Perhaps most closely related to our work is Zipt [8], which enables comparative user performance testing at scale.…”

Section: Related Workmentioning

confidence: 99%

Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning

Swearngin

2019

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Mobile UI Corpus--------- Model Labeled ScreensPredictions Train Requests 82% Figure 1: Our deep model learns from a large-scale dataset of mobile tappability collected via crowdsourcing. It predicts tappability of interface elements and identifies mismatches between designer intention and user perception, and is served in the TapShoe tool that can help designers and developers to uncover potential usability issues about their mobile interfaces. ABSTRACTTapping is an immensely important gesture in mobile touchscreen interfaces, yet people still frequently are required to learn which elements are tappable through trial and error. Predicting human behavior for this everyday gesture can help mobile app designers understand an important aspect of the usability of their apps without having to run a user study. In this paper, we present an approach for modeling tappability of mobile interfaces at scale. We conducted largescale data collection of interface tappability over a rich set of mobile apps using crowdsourcing and computationally investigated a variety of signifiers that people use to distinguish tappable versus not-tappable elements. Based on the dataset, we developed and trained a deep neural network that predicts how likely a user will perceive an interface * This work was completed while the author was an intern at Google. element as tappable versus not tappable. Using the trained tappability model, we developed TapShoe, a tool that automatically diagnoses mismatches between the tappability of each element as perceived by a human user-predicted by our model, and the intended or actual tappable state of the element specified by the developer or designer. Our model achieved reasonable accuracy: mean precision 90.2% and recall 87.0%, in matching human perception on identifying tappable UI elements. The tappability model and TapShoe were well received by designers via an informal evaluation with 7 professional interaction designers. CCS CONCEPTS• Human-centered computing → Interactive systems and tools; Systems and tools for interaction design.

show abstract

Section: Related Workmentioning

confidence: 99%

Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning

Swearngin

2019

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…Today, this assumption seems reasonable, given the high adoption rate of smartphones [11]. Most app GUIs are designed to be easy-to-use with common design patterns [18], so users are likely able to understand the screens even if they have never used the particular app before. Using app GUIs is still by far the most common means through which the users access computing services.…”

Section: Discussionmentioning

confidence: 99%

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

Chen

Xia

et al. 2020

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

A major problem in task-oriented conversational agents is the lack of support for the repair of conversational breakdowns. Prior studies have shown that current repair strategies for these kinds of errors are often ineffective due to: (1) the lack of transparency about the state of the system's understanding of the user's utterance; and (2) the system's limited capabilities to understand the user's verbal attempts to repair natural language understanding errors. This paper introduces SOVITE, a new multi-modal (speech plus direct manipulation) interface that helps users discover, identify the causes of, and recover from conversational breakdowns using the resources of existing mobile app GUIs for grounding. SOVITE displays the system's understanding of user intents using GUI screenshots, allows users to refer to third-party apps and their GUI screens in conversations as inputs for intent disambiguation, and enables users to repair breakdowns using direct manipulation on these screenshots. The results from a remote user study with 10 users using SOVITE in 7 scenarios suggested that SOVITE's approach is usable and effective.

show abstract

“…Swire is able to retrieve registration (Example a) and 'closing menu' (Example b) fows that are commonly implemented by designers. Since Rico also contain transition details between each consequent UIs, these examples can demonstrate popular animation patterns [6] that provide inspiration to interaction and animation designers.…”

Section: User Flow Examplesmentioning

confidence: 99%

“…Design examples are commonly curated and collected by designers to explore the design space and inspire new design ideas. The value brought by these examples has led to the creation of several large-scale mobile UI repositories, such as UXArchive [1], ERICA [6] and Rico [5]. Swire is developed by collecting a dataset of sketches from designers based on designs found in Rico.…”

Section: Related Work Mobile User Interface Repositoriesmentioning

confidence: 99%

“…The recent introduction of data-driven design and the emergence of large-scale user interface (UI) datasets provide designers with large corpuses of real examples that refect multiple facets of successful designs [11]. These examples embody rich information such as popular visual illustrations, common fow patterns and high-fdelity layout implementations [6] that can greatly augment various design tasks [12,18,25].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation