The Parallels between the Study of Crossmodal Correspondence and the Design of Cross-Sensory Mappings

ACM Trans. Multimedia Comput. Commun. Appl.

Hussain

Covaci

et al. 2019

Different senses provide us with information of various levels of precision and enable us to construct a more precise representation of the world. Rich multisensory simulations are thus beneficial for comprehension, memory reinforcement or retention of information. Crossmodal mappings refer to the systematic associations often made between different sensory modalities (e.g., high pitch is matched with angular shapes) and govern multisensory processing. A great deal of research effort has been put in exploring crossmodal correspondences in the field of cognitive science. However, the possibilities they open in the digital world have been relatively unexplored. Mulsemedia-multiple sensorial media-provides a highly immersive experience to the users and enhances their Quality of Experience (QoE) in the digital world. Thus, we consider that studying the plasticity and the effects of crossmodal correspondences in a mulsemedia setup can bring interesting insights about improving the human computer dialogue and experience. In our experiments, we exposed users to videos with certain visual dimensions (brightness, color and shape) and we investigated if the pairing with a crossmodal matching sound (high and low pitch) and the corresponding auto-generated vibrotactile effects (produced by a haptic vest) lead to an enhanced QoE. For this, we captured the eye gaze and the heart rate of users while experiencing mulsemedia and we asked them to fill in a set of questions targeting their enjoyment and perception at the end of the experiment. Results showed differences in eye gaze patterns and heart rate between the experimental and the control group indicating changes in participants' engagement when videos were accompanied by matching crossmodal sounds (this effect was the strongest for the video displaying angular shapes and high pitch audio) and transitively generated crossmodal vibrotactile effects. CCS Concepts: • Human-centered computing → HCI design and evaluation methods;Interaction paradigms;Interaction techniques;Empirical studies in HCI;

Section: Auditory-visual Crossmodal Correspondences Research In Compumentioning

confidence: 98%

Using Eye Tracking and Heart-Rate Activity to Examine Crossmodal Correspondences QoE in Mulsemedia

ACM Trans. Multimedia Comput. Commun. Appl.

Hussain

Covaci

et al. 2019

“…As shown in the previous section, crossmodal correspondences between different sensory dimensions show an interesting practical potential. However, the work on this outside the field of cognitive sciences is limited, notwithstanding the fact that Understanding cognitive processes across all sensory modalities could play an important role in developing efficient digital multimodal interfaces that consider all the subtleties involved in human perception [50].…”

Section: B Olfactory-visual Crossmodal Correspondences Applications mentioning

confidence: 99%

How Do We Experience Crossmodal Correspondent Mulsemedia Content?

Covaci

Saleme

et al. 2020

IEEE Trans. Multimedia

“…As can be seen, studies on crossmodal correspondences research in computer graphics and human-computer interaction provide insights about sensory replacement/combination under different circumstances. These mappings have a promising potential in designing interfaces and displays that tap into a user's mental model [72]. Thus, we believe that crossmodal mappings could reveal insightful information in other contexts to help to understand the users' perception and therefore improve human-computer interaction.…”

Section: Auditory-visual and Olfactory-visual Crossmodal Correspondenmentioning

confidence: 99%

QoE of cross-modally mapped Mulsemedia: an assessment using eye gaze and heart rate

Hussain

Kani‐Zabihi

et al. 2020

Multimed Tools Appl

A great deal of research effort has been put in exploring crossmodal correspondences in the field of cognitive science which refer to the systematic associations frequently made between different sensory modalities (e.g. high pitch is matched with angular shapes). However, the possibilities cross-modality opens in the digital world have been relatively unexplored. Therefore, we consider that studying the plasticity and the effects of crossmodal correspondences in a mulsemedia setup can bring novel insights about improving the human-computer dialogue and experience. Mulsemedia refers to the combination of three or more senses to create immersive experiences. In our experiments, users were shown six video clips associated with certain visual features based on color, brightness, and shape. We examined if the pairing with crossmodal matching sound and the corresponding auto-generated haptic effect, and smell would lead to an enhanced user QoE. For this, we used an eye-tracking device as well as a heart rate monitor wristband to capture users' eye gaze and heart rate whilst they were experiencing mulsemedia. After each video clip, we asked the users to complete an on-screen questionnaire with a set of questions related to smell, sound and haptic effects targeting their enjoyment and perception of the experiment. Accordingly, the eye gaze and heart rate results showed significant influence of the cross-modally mapped multisensorial effects on the users' QoE. Our results highlight that when the olfactory content is crossmodally congruent with the visual content, the visual attention of the users seems shifted towards the correspondent visual feature. Crosmodally matched media is also shown to result in an enhanced QoE compared to a video only condition.