Beyond MAYA for game-changing multisensory design

Rousi, Rebekah; Silvennoinen, Johanna; Perälä, Piia; Jokinen, Jussi

doi:10.1145/3131085.3131113

Cited by 2 publications

(1 citation statement)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Positive emotional outcome is dependent on the context of the design and its appraisal is strongly connected to multisensory integration. Expectations have an important role in HCI, thus crossmodal correspondences could be one of the underlying dynamics of a positive experience [53]. As can be seen, studies on crossmodal correspondences research in computer graphics and human-computer interaction provide insights about sensory replacement/combination under different circumstances.…”

Section: Auditory-visual and Olfactory-visual Crossmodal Correspondenmentioning

confidence: 99%

QoE of cross-modally mapped Mulsemedia: an assessment using eye gaze and heart rate

Mesfin

Hussain

Kani‐Zabihi

et al. 2020

Multimed Tools Appl

View full text Add to dashboard Cite

A great deal of research effort has been put in exploring crossmodal correspondences in the field of cognitive science which refer to the systematic associations frequently made between different sensory modalities (e.g. high pitch is matched with angular shapes). However, the possibilities cross-modality opens in the digital world have been relatively unexplored. Therefore, we consider that studying the plasticity and the effects of crossmodal correspondences in a mulsemedia setup can bring novel insights about improving the human-computer dialogue and experience. Mulsemedia refers to the combination of three or more senses to create immersive experiences. In our experiments, users were shown six video clips associated with certain visual features based on color, brightness, and shape. We examined if the pairing with crossmodal matching sound and the corresponding auto-generated haptic effect, and smell would lead to an enhanced user QoE. For this, we used an eye-tracking device as well as a heart rate monitor wristband to capture users' eye gaze and heart rate whilst they were experiencing mulsemedia. After each video clip, we asked the users to complete an on-screen questionnaire with a set of questions related to smell, sound and haptic effects targeting their enjoyment and perception of the experiment. Accordingly, the eye gaze and heart rate results showed significant influence of the cross-modally mapped multisensorial effects on the users' QoE. Our results highlight that when the olfactory content is crossmodally congruent with the visual content, the visual attention of the users seems shifted towards the correspondent visual feature. Crosmodally matched media is also shown to result in an enhanced QoE compared to a video only condition.

show abstract

Section: Auditory-visual and Olfactory-visual Crossmodal Correspondenmentioning

confidence: 99%