Effects of auditory, haptic and visual feedback on performing gestures by gaze or by hand

Abstract

Modern interaction techniques like non-intrusive gestures provide means for interacting with distant displays and smart objects without touching them. We were interested in the effects of feedback modality (auditory, haptic or visual) and its combined effect with input modality on user performance and experience in such interactions. Therefore, we conducted two exploratory experiments where numbers were entered, either by gaze or hand, using gestures composed of four stroke elements (up, down, left and right). In Experiment 1, a simple feedback was given on each stroke during the motor action of gesturing: an audible click, a haptic tap or a visual flash. In Experiment 2, a semantic feedback was given on the final gesture: the executed number was spoken, coded by haptic taps or shown as text. With simultaneous simple feedback in Experiment 1, performance with hand input was slower but more accurate than with gaze input. With semantic feedback in Experiment 2, however, hand input was only slower. Effects of feedback modality were of minor importance; nevertheless, semantic haptic feedback in Experiment 2 showed to be useless at least without extensive training. Error patterns differed between both input modes, but again not dependent on feedback modality. Taken together, the results show that in designing gestural systems, choosing a feedback modality can be given a low priority; it can be chosen according to the task, context and user preferences. ARTICLE HISTORY

Evaluating the Effects of Feedback Type on Older Adults' Performance in Mid-Air Pointing and Target Selection

“…There are many ageing factors that compromise the pointing abilities of older users. Muscle strength begins to gradually decline from the age of 50 [18], leading to easy fatiguing, decrease in motor control, limited range of motion and slower reaction times [8,24]. As a matter of comparison, an older adult at the age of 90 is expected to have a range of motion that is only 60% of the range of motion of an average 30 years old individual [8,18].…”

“…Muscle strength begins to gradually decline from the age of 50 [18], leading to easy fatiguing, decrease in motor control, limited range of motion and slower reaction times [8,24]. As a matter of comparison, an older adult at the age of 90 is expected to have a range of motion that is only 60% of the range of motion of an average 30 years old individual [8,18]. Furthermore, continuous use of hand and arm movements for pointing in mid-air, without proper and frequent relaxation of the arms, can lead older users to physical tiredness and may largely impact accuracy and steadiness of movements, performance consistency, and user experience throughout the interaction [7,24,33].…”

“…Reduced visual perception is a main contributing factor on how older users perceive and interact with different interfaces, specifically touchless interfaces, since most interaction methods rely heavily on visual feedback [6,11]. Older adults may face greater difficulties at following cursor movements, locating targets on screen, and perceiving GUI changes due to limited visual acuity, which can also affect reaction times once they make a gesture [18,20,30]. Aged hearing function and reduced sensitivity to sound [18] can also affect the effectiveness of sound cues and audio feedback in cases where these are provided.…”

“…Older adults may face greater difficulties at following cursor movements, locating targets on screen, and perceiving GUI changes due to limited visual acuity, which can also affect reaction times once they make a gesture [18,20,30]. Aged hearing function and reduced sensitivity to sound [18] can also affect the effectiveness of sound cues and audio feedback in cases where these are provided. Despite limited haptics in mid-air, reduced tactile sensitivity and acuity may also contribute to poorer "point-and-select" performance.…”

“…Older adults are known to experience age-related decline in motor, cognitive and sensory abilities that can affect their daily tasks, including the way they interact with technology [3,8,24]. Older adults experience natural changes in motor dexterity and muscular strength that may affect pointing abilities due to loss of hand mobility and decreased range of motion [18,30]. If these physical limitations are not anticipated by the system design, then older users may come across failed interaction attempts, leading to frustrating interaction experiences and further hindrances to technology use.…”