Analysis and Modeling of Grid Performance on Touchscreen Mobile Devices

Abstract

Touchscreen mobile devices can afford rich interaction behaviors but they are complex to model. Scrollable twodimensional grids are a common user interface on mobile devices that allow users to access a large number of items on a small screen by direct touch. By analyzing touch input and eye gaze of users during grid interaction, we reveal how multiple performance components come into play in such a task, including navigation, visual search and pointing. These findings inspired us to design a novel predictive model that combines these components for modeling grid tasks. We realized these model components by employing both traditional analytical methods and data-driven machine learning approaches. In addition to showing high accuracy achieved by our model in predicting human performance on a test dataset, we demonstrate how such a model can lead to a significant reduction in interaction time when used in a predictive user interface.

Modeling Angle-Based Pointing Tasks in Augmented Reality Interfaces

“…The complexity of the calculation task can be used to evaluate the difficulty of the related task. To date, researchers have proposed many quantitative models for different interactive tasks and interactive devices based on Fitts' law and its existing extensions, such as the AR pointing model [8,10], scrolling interaction [19,22], touch interaction [4,17], the manipulation task [15], peephole pointing [7,20], menu selection [25], visual search [23] and head interaction [5,16].…”

“…With the widespread use of touch screen mobile devices, a scrollable twodimensional grid has become a universal user interface on mobile devices. Pfeuffer [22] proposed a novel prediction model to model grid tasks on touch screen mobile devices. In terms of influencing factors of task performance, they considered factors such as navigation, visual search and pointing, and they used machine learning methods to study the interactions between these.…”

“…The development of mobile computing devices, such as mobile phones and tablet computers, which have powerful computing capacities, is becoming ubiquitous. However, due to physical screen size limitations [7,8,19,22,24], the traditional interaction mode is not natural enough to meet people's needs. Augmented reality (AR) is a technology that superimposes real environments and virtual objects on the same screen or space in real time.…”

How We Type: Eye and Finger Movement Strategies in Mobile Typing

“…On internet-based tasks on tablets, gaze has been observed to precede touch with similar spatial and temporal features as observed with the mouse, but with individual differences [60,61]. Interaction between gaze and touch has also been utilized to adapt UIs with the help of a predictive model [49]. On tablets with split keyboards, it is often enough to attend only the text entry area and use peripheral vision to guide fingers [40].…”

Optimizing User Interface Layouts via Gradient Descent

“…They range from evaluative metrics [2] and models of human performance [14,3] that designers can use to assess their designs, to tools and techniques that can optimize aspects of the design [26,23,7,24]. Tremendous progress has been made in modeling human performance on interaction tasks, and recently, deep learning approaches have been introduced, specifically for modelling menu item selection [14] and selecting items in a grid-based interface [21].…”