Crossmodal Audio and Tactile Interaction with Mobile Touchscreens

Abstract

Touchscreen mobile devices often use cut-down versions of desktop user interfaces placing high demands on the visual sense that may prove awkward in mobile settings. The research in this thesis addresses the problems encountered by situationally impaired mobile users by using crossmodal interaction to exploit the abundant similarities between the audio and tactile modalities. By making information available to both senses, users can receive the information in the most suitable way, without having to abandon their primary task to look at the device.This thesis begins with a literature review of related work followed by a definition of crossmodal icons. Two icons may be considered to be crossmodal if and only if they provide a common representation of data, which is accessible interchangeably via different modalities. Two experiments investigated possible parameters for use in crossmodal icons with resultsshowing that rhythm, texture and spatial location are effective.A third experiment focused on learning multi-dimensional crossmodal icons and the extent to which this learning transfers between modalities. The results showed identification rates of 92% for three-dimensional audio crossmodal icons when trained in the tactile equivalents, and identification rates of 89% for tactile crossmodal icons when trained in the audio equivalent.Crossmodal icons were then incorporated into a mobile touchscreen QWERTY keyboard.Experiments showed that keyboards with audio or tactile feedback produce fewer errors and greater speeds of text entry compared to standard touchscreen keyboards. The next study examined how environmental variables affect user performance with the same keyboard. The data showed that each modality performs differently with varying levels of background noise or vibration and the exact levels at which these performance decreases occur were established.The final study involved a longitudinal evaluation of a touchscreen application, CrossTrainer, focusing on longitudinal effects on performance with audio and tactile feedback, the impact of context on performance and personal modality preference. The results show that crossmodal audio and tactile icons are a valid method of presenting information to situationally impaired mobile touchscreen users with recognitions rates of 100% over time.This thesis concludes with a set of guidelines on the design and application of crossmodal audio and tactile feedback to enable application and interface designers to employ such feedback in all systems.

Investigating the Affective Impression of Tactile Feedback on Mobile Devices

“…However, Pattern 6 was rated significantly more arousing than Pattern 15 and 16 even though its duration is lower. This shows that rhythm is one very important feature, following the findings of earlier research (Hoggan 2010). Unsurprisingly, it is possible to create annoying tactile feedback messages on the mobile device if they exceed a certain duration and intensity.…”

“…Going beyond unimodal tactile feedback, also studies with crossmodal feedback have been done. Hoggan (2010) found out that crossmodal audiotactile icons are helpful in different usage situations of mobile touchscreen devices, and derived a number of guidelines on the design and application of crossmodal feedback.…”

Emotional Feedback for Mobile Devices

“…However, Pattern 6 was rated significantly more arousing than Pattern 15 and Pattern 16 even though its duration is lower. This shows that rhythm is one very important feature, following the findings of earlier research [75]. Unsurprisingly, it is possible to create annoying tactile feedback messages on the mobile device if they exceed a certain duration and intensity.…”

“…by Hoggan and colleagues [71,73]. They found audiotactile icons being helpful in different usage situations of mobile touchscreen devices, and derived a number of guidelines on the design and application of crossmodal feedback [75].…”

Exploring crossmodal perceptual enhancement and integration in a sequence-reproducing task with cognitive priming

“…In the HCI domain, crossmodal information has been applied in various multisensory design situations, using both bottom-up and top-down approaches, with the research emphasis on robust information perception [35,21,17], sensory substitution [29], sports skill acquisition [42,43,41] ,embodied interaction experience [4], and data representation through multisensory modalities [53,30,19]. Interestingly, some applications have been designed where congruent crossmodal associations were not always in line with intended activity.…”