Learning Design Semantics for Mobile Apps

Liu, Thomas F.; Craft, Mark; Situ, Jason; Yumer, Ersin; Mĕch, Radomír; Kumar, Ranjitha

doi:10.1145/3242587.3242650

Cited by 120 publications

(76 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, urgency can be created by a blinking timer; similarly, Hidden Subscriptions can make the default option (e.g., subscribing to a paid service) visually more appealing and noticeable than its alternative (e.g., not subscribing). One starting point to detect such interfaces could be to incorporate style and color as features for clustering, or even use the design mining literature [39,56,59] to analyze specific types of interfaces (e.g., page headers) in isolation. Finally, researchers can leverage our descriptive taxonomy of dark pattern characteristics to study and analyze dark patterns in other domains, such as emails and mobile applications.…”

Section: Dark Patterns and Future Studies At Scalementioning

confidence: 99%

Dark Patterns at Scale

Mathur

Acar

Friedman

et al. 2019

Proc. ACM Hum.-Comput. Interact.

256

View full text Add to dashboard Cite

Dark patterns are user interface design choices that benefit an online service by coercing, steering, or deceiving users into making unintended and potentially harmful decisions. We present automated techniques that enable experts to identify dark patterns on a large set of websites. Using these techniques, we study shopping websites, which often use dark patterns to influence users into making more purchases or disclosing more information than they would otherwise. Analyzing ∼53K product pages from ∼11K shopping websites, we discover 1,818 dark pattern instances, together representing 15 types and 7 broader categories. We examine these dark patterns for deceptive practices, and find 183 websites that engage in such practices. We also uncover 22 third-party entities that offer dark patterns as a turnkey solution. Finally, we develop a taxonomy of dark pattern characteristics that describes the underlying influence of the dark patterns and their potential harm on user decision-making. Based on our findings, we make recommendations for stakeholders including researchers and regulators to study, mitigate, and minimize the use of these patterns.

show abstract

Section: Dark Patterns and Future Studies At Scalementioning

confidence: 99%

Dark Patterns at Scale

Mathur

Acar

Friedman

et al. 2019

Proc. ACM Hum.-Comput. Interact.

256

View full text Add to dashboard Cite

show abstract

“…We also manually segmented and labeled UI elements on the screenshots to form an element taxonomy. For this, we followed existing design guidelines and previous research [1,16,34,41,49]. The taxonomy was determined by two human coders, through a consensus-driven, iterative process.…”

Section: Methodsmentioning

confidence: 99%

Understanding Visual Saliency in Mobile User Interfaces

Leiva

Xue

Bansal

et al. 2020

22nd International Conference on Human-Computer Interaction With Mobile Devices and Services

View full text Add to dashboard Cite

For graphical user interface (UI) design, it is important to understand what attracts visual attention. While previous work on saliency has focused on desktop and web-based UIs, mobile app UIs differ from these in several respects. We present findings from a controlled study with 30 participants and 193 mobile UIs. The results speak to a role of expectations in guiding where users look at. Strong bias toward the top-left corner of the display, text, and images was evident, while bottom-up features such as color or size affected saliency less. Classic, parameter-free saliency models showed a weak fit with the data, and data-driven models improved significantly when trained specifically on this dataset (e.g., NSS rose from 0.66 to 0.84). We also release the first annotated dataset for investigating visual saliency in mobile UIs. CCS CONCEPTS • Human-centered computing → Empirical studies in ubiquitous and mobile computing; • Computing methodologies → Computer vision.

show abstract

“…This com bination of multi-modal and mixed-initiative approaches has been previously applied in the interactive task learning process in systems such as [1,30,36,39]. SOVITE bridges an important gap in these systems, as they focus on the ambiguities, uncer tainties, and vagueness embedded in the user instructions of In terms of the technique used, SOVITE extracts the semantics of app GUIs [13,43] for grounding natural language conver sations. Compared with the previous systems that used the semantics of app GUIs for learning new tasks [36,38,58], extracting task flows [40], and supporting invoking individual GUI widgets with voice commands [64], a new idea in SO VITE is that it encodes app GUIs into the same vector space as natural language utterances, allowing the system to look up semantically relevant task intents when the user refers to apps and app GUI screens in the dialogues for repairing intent detection errors (details in the Implementation section).…”

Section: Multi-modal Mixed-initiative Disambiguation Interfacesmentioning

confidence: 99%

“…SOVITE's current mechanism only takes the text labels shown on GUIs into consideration. In the future, we plan to capture more comprehensive semantics of app GUI screens by leveraging the GUI layouts (e.g., the distance be tween elements [41] and design patterns [18,43]), control flows among GUI screens [40], and large collections of user interaction traces. The availability of large-scale GUI datasets like RICO [17] makes future experiments in this area feasible.…”

Section: Limitations and Future Workmentioning

confidence: 99%

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

Chen

Xia

et al. 2020

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

A major problem in task-oriented conversational agents is the lack of support for the repair of conversational breakdowns. Prior studies have shown that current repair strategies for these kinds of errors are often ineffective due to: (1) the lack of transparency about the state of the system's understanding of the user's utterance; and (2) the system's limited capabilities to understand the user's verbal attempts to repair natural language understanding errors. This paper introduces SOVITE, a new multi-modal (speech plus direct manipulation) interface that helps users discover, identify the causes of, and recover from conversational breakdowns using the resources of existing mobile app GUIs for grounding. SOVITE displays the system's understanding of user intents using GUI screenshots, allows users to refer to third-party apps and their GUI screens in conversations as inputs for intent disambiguation, and enables users to repair breakdowns using direct manipulation on these screenshots. The results from a remote user study with 10 users using SOVITE in 7 scenarios suggested that SOVITE's approach is usable and effective.

show abstract

Learning Design Semantics for Mobile Apps

Cited by 120 publications

References 15 publications

Dark Patterns at Scale

Dark Patterns at Scale

Understanding Visual Saliency in Mobile User Interfaces

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

Contact Info

Product

Resources

About