Towards Multimodal Affective Feedback

Abstract

We explored how emotional cues presented in visual and haptic modalities interact. We constructed an affective haptic dataset, and used the emotional visual stimuli from the International Affective Picture System (IAPS). Participants were asked to rate the visual stimuli, haptic stimuli and visualhaptic stimuli. Analysis of the results indicated that the presence of haptic stimulus affects the arousal of the visual stimulus, but does not affect the valence significantly. We further explored this interaction in terms of the intensity, frequency, waveform and rhythm of the haptic stimuli. Finally, we generated a set of guidelines on visual-haptic interaction that could be used to design multimodal affective feedback.

Multi-moji

“…There is also a roughly even number and spread of stimuli in each of the four quadrants, although the bottom-left remains slightly sparser. These results are significant and promising, as the top-right and bottom-left quadrants have proven difficult to access in other research [3,49,51,53], and no other research has found such an even spread of stimuli throughout the model.…”

“…Our multimodal combinations, or multi-moji, have led to a wider range of conveyable emotional states than individual modalities are capable of, mostly in the top-right and bottomleft quadrants of the valence-arousal model [31], which have traditionally been the areas most poorly covered by feedback in HCI [3,49,51,53]. This means that UI designers now have a much richer set of possible emotional cues to use, to expand the affective range of feedback.…”

“…In summary, short or rapid vibrotactile pulses, or longer overall patterns, are viewed as highly arousing [27,36,53]. The carrier frequency and intensity of the vibration both influence V and A simultaneously, with increases in either parameter increasing both V and A [3,27,36,53]. Producing vibrations of different texture, e.g., by increasing "roughness" [36] increases A and decreases V [53].…”

“…Salminen et al [32] found that combining positive and negative speech samples with the vibrotactile waveform of that speech made it seem more arousing and dominating. Akshita et al [3] found that vibration made IAPS images [22] more arousing, but did not influence their valence.…”

“…Some research has studied the modulation of an emotional response to one modality, through the simultaneous presentation of another [2,3,16,32]. However, these studies only ask participants to focus on and rate the one modality, with those ratings potentially altered by the second.…”