QoE of cross-modally mapped Mulsemedia: an assessment using eye gaze and heart rate

Mesfin, Gebremariam; Hussain, Nadia; Kani‐Zabihi, Elahe; Covaci, Alexandra; Saleme, Estêvão Bissoli; Ghinea, Gheorghiţă

doi:10.1007/s11042-019-08473-5

Cited by 8 publications

(8 citation statements)

References 69 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Furthermore, they did find significant associations between their shapes and odors (lemon and vanilla) and an emotional dimension (arousal). Mesfin et al [48] investigated the use of crossmodally congruent olfactory stimuli in multisensory multimedia. Their results revealed that crossmodally matched media enhances the quality of experience compared to a video only condition.…”

Section: Discussionmentioning

confidence: 99%

Smelling sensations: Olfactory crossmodal correspondences (JPI-first)

Ward

Wuerger

Marshall

2022

View full text Add to dashboard Cite

Olfaction is ingrained into the fabric of our daily lives and constitutes an integral part of our perceptual reality. Within this reality, there are crossmodal interactions and sensory expectations; understanding how olfaction interacts with other sensory modalities is crucial for augmenting interactive experiences with more advanced multisensorial capabilities. This knowledge will eventually lead to better designs, more engaging experiences, and enhancing the perceived quality of experience. Toward this end, the authors investigated a range of crossmodal correspondences between ten olfactory stimuli and different modalities (angularity of shapes, smoothness of texture, pleasantness, pitch, colors, musical genres, and emotional dimensions) using a sample of 68 observers. Consistent crossmodal correspondences were obtained in all cases, including our novel modality (the smoothness of texture). These associations are most likely mediated by both the knowledge of an odor’s identity and the underlying hedonic ratings: the knowledge of an odor’s identity plays a role when judging the emotional and musical dimensions but not for the angularity of shapes, smoothness of texture, perceived pleasantness, or pitch. Overall, hedonics was the most dominant mediator of crossmodal correspondences.

show abstract

Section: Discussionmentioning

confidence: 99%

Smelling sensations: Olfactory crossmodal correspondences (JPI-first)

Ward

Wuerger

Marshall

2022

View full text Add to dashboard Cite

show abstract

“…Another future direction that could be addressed in the context of authoring tools is that of crossmodal correspondences [61], which may affect our perceptual experiences. This concept refers to a compatibility effect between attributes or dimensions of a stimulus in different sensory modalities.…”

Section: Future Directionsmentioning

confidence: 99%

Beyond Multimedia Authoring

2021

Self Cite

View full text Add to dashboard Cite

The mulsemedia (Multiple Sensorial Media (MulSeMedia)) concept has been explored to provide users with new sensations using other senses beyond sight and hearing. The demand for producing such applications has motivated various studies in the mulsemedia authoring phase. To encourage researchers to explore new solutions for enhancing the mulsemedia authoring, this survey article reviews several mulsemedia authoring tools and proposals for representing sensory effects and their characteristics. The article also outlines a set of desirable features for mulsemedia authoring tools. Additionally, a multimedia background is discussed to support the proposed study in the mulsemedia field. Open challenges and future directions regarding the mulsemedia authoring phase are also discussed.

show abstract

“…A correlation was found between these objective metrics with the arousal and subjective ratings of QoE [ 32 ]. Eye gaze data and heart rate have been analyzed for evaluating the enjoyment and perception of viewers while experiencing mulsemedia content [ 33 , 34 ]. Cross-modal correspondences were also identified when mapped with multi-sensory effects.…”

Section: Introductionmentioning

confidence: 99%

Physiological Sensors Based Emotion Recognition While Experiencing Tactile Enhanced Multimedia

Raheel

Majid

Alnowami

et al. 2020

Sensors

View full text Add to dashboard Cite

Emotion recognition has increased the potential of affective computing by getting an instant feedback from users and thereby, have a better understanding of their behavior. Physiological sensors have been used to recognize human emotions in response to audio and video content that engages single (auditory) and multiple (two: auditory and vision) human senses, respectively. In this study, human emotions were recognized using physiological signals observed in response to tactile enhanced multimedia content that engages three (tactile, vision, and auditory) human senses. The aim was to give users an enhanced real-world sensation while engaging with multimedia content. To this end, four videos were selected and synchronized with an electric fan and a heater, based on timestamps within the scenes, to generate tactile enhanced content with cold and hot air effect respectively. Physiological signals, i.e., electroencephalography (EEG), photoplethysmography (PPG), and galvanic skin response (GSR) were recorded using commercially available sensors, while experiencing these tactile enhanced videos. The precision of the acquired physiological signals (including EEG, PPG, and GSR) is enhanced using pre-processing with a Savitzky-Golay smoothing filter. Frequency domain features (rational asymmetry, differential asymmetry, and correlation) from EEG, time domain features (variance, entropy, kurtosis, and skewness) from GSR, heart rate and heart rate variability from PPG data are extracted. The K nearest neighbor classifier is applied to the extracted features to classify four (happy, relaxed, angry, and sad) emotions. Our experimental results show that among individual modalities, PPG-based features gives the highest accuracy of 78.57 % as compared to EEG- and GSR-based features. The fusion of EEG, GSR, and PPG features further improved the classification accuracy to 79.76 % (for four emotions) when interacting with tactile enhanced multimedia.

show abstract

QoE of cross-modally mapped Mulsemedia: an assessment using eye gaze and heart rate

Cited by 8 publications

References 69 publications

Smelling sensations: Olfactory crossmodal correspondences (JPI-first)

Smelling sensations: Olfactory crossmodal correspondences (JPI-first)

Beyond Multimedia Authoring

Physiological Sensors Based Emotion Recognition While Experiencing Tactile Enhanced Multimedia

Contact Info

Product

Resources

About