Rico

Abstract

Data-driven models help mobile app designers understand best practices and trends, and can be used to make predictions about design performance and support the creation of adaptive UIs. This paper presents Rico, the largest repository of mobile app designs to date, created to support five classes of data-driven applications: design search, UI layout generation, UI code generation, user interaction modeling, and user perception prediction. To create Rico, we built a system that combines crowdsourcing and automation to scalably mine design and interaction data from Android apps at runtime. The Rico dataset contains design data from more than 9.7k Android apps spanning 27 categories. It exposes visual, textual, structural, and interactive design properties of more than 72k unique UI screens. To demonstrate the kinds of applications that Rico enables, we present results from training an autoencoder for UI layout similarity, which supports queryby-example search over UIs.

Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning

“…To identify tappability issues automatically, we needed to collect data on a large scale to allow us to use a machine learning approach for this problem. Recently, data-driven approaches have been used to identify usability issues [8], and collect mobile app design data at scale [7,9]. Perhaps most closely related to our work is Zipt [8], which enables comparative user performance testing at scale.…”

“…We designed a crowdsourcing task to simulate a tappability study across a large corpus of Android mobile apps [7], using the interface shown in Figure 2. The left side of the interface displayed a mobile app screenshot.…”

“…To collect our tappability dataset, we selected a set of 3,470 unique, randomly chosen screens from the Rico dataset [7] and had crowd workers label elements randomly sampled from these screens as either tappable or not tappable. We selected the elements for the workers to label in the following manner.…”

“…Several element types have conventions for visual appearance, thus users would consistently perceive them as tappable [23] (e.g., buttons). We examined how accurately workers label each interface element type from a subset of Android class types in the Rico dataset [7]. shows the distribution of tappable and not-tappable elements by type labeled by human workers.…”

Swire

“…This paper ofers two major contributions. First, we collected the frst large-scale sketch dataset consisting of 3802 sketches corresponding to 2201 UI examples from the Rico dataset [5] drawn by experienced UI designers recruited on an online freelance work platform. This dataset allows us to develop techniques capable of learning UI sketch patterns and supports future work in this area.…”