An end-to-end tool chain for Sensory Experience based on MPEG-V

Waltl, Markus; Rainer, Benjamin; Timmerer, Christian; Hellwagner, Hermann

doi:10.1016/j.image.2012.10.009

Cited by 63 publications

(39 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…a virtual 3D space in a game) [56]. Many tools have been developed to aid this process, such as SEVino [75], SMURF [32], RoSE Studio [5], and Real 4D studio [59]. The works of Kim et al [33] and Oh and Huh [48] are endeavors to automatically produce mulsemedia metadata.…”

Section: Mulsemedia and Qoementioning

confidence: 99%

“…Following that, the mulsemedia effects can be encoded for transport, processed and emitted for distribution to providers, distributed to the end-users and then decoded by systems, and finally, rendered by different devices, which in turn, will deliver them to the end users. Mulsemedia players and renderers to be used with other multimedia applications have also been created to reproduce and deliver mulsemedia experiences, notably SEMP [75] and PlaySEM [55], which are open-source. A mulsemedia system entails weaving multiple technologies to connect different entities, distribute the sensory signals, and render sensory effects appropriately Saleme et al [56].…”

Section: Mulsemedia and Qoementioning

confidence: 99%

See 1 more Smart Citation

QoE of cross-modally mapped Mulsemedia: an assessment using eye gaze and heart rate

Mesfin

Hussain

Kani‐Zabihi

et al. 2020

Multimed Tools Appl

View full text Add to dashboard Cite

A great deal of research effort has been put in exploring crossmodal correspondences in the field of cognitive science which refer to the systematic associations frequently made between different sensory modalities (e.g. high pitch is matched with angular shapes). However, the possibilities cross-modality opens in the digital world have been relatively unexplored. Therefore, we consider that studying the plasticity and the effects of crossmodal correspondences in a mulsemedia setup can bring novel insights about improving the human-computer dialogue and experience. Mulsemedia refers to the combination of three or more senses to create immersive experiences. In our experiments, users were shown six video clips associated with certain visual features based on color, brightness, and shape. We examined if the pairing with crossmodal matching sound and the corresponding auto-generated haptic effect, and smell would lead to an enhanced user QoE. For this, we used an eye-tracking device as well as a heart rate monitor wristband to capture users' eye gaze and heart rate whilst they were experiencing mulsemedia. After each video clip, we asked the users to complete an on-screen questionnaire with a set of questions related to smell, sound and haptic effects targeting their enjoyment and perception of the experiment. Accordingly, the eye gaze and heart rate results showed significant influence of the cross-modally mapped multisensorial effects on the users' QoE. Our results highlight that when the olfactory content is crossmodally congruent with the visual content, the visual attention of the users seems shifted towards the correspondent visual feature. Crosmodally matched media is also shown to result in an enhanced QoE compared to a video only condition.

show abstract

Section: Mulsemedia and Qoementioning

confidence: 99%

Section: Mulsemedia and Qoementioning

confidence: 99%

QoE of cross-modally mapped Mulsemedia: an assessment using eye gaze and heart rate

Mesfin

Hussain

Kani‐Zabihi

et al. 2020

Multimed Tools Appl

View full text Add to dashboard Cite

show abstract

“…To do so, the museum will be using a mulsemedia platform called PlaySEM [28], which has a video player and a renderer that support sensory effects. The video-clips are annotated with sensory effect descriptions in MPEG-V (with the help of SEVino [34]), that include light, vibration, spraying, kinesthetic, tactile, wind, scent, taste, fog, and temperature effects. Visual media will be displayed on screens, projectors, head-mounted displays.…”

Section: Hypothetical Mulsemedia Scenariomentioning

confidence: 99%

Towards a reference ontology on mulsemedia systems

Saleme

Santos

Falbo

et al. 2018

Proceedings of the 10th International Conference on Management of Digital EcoSystems

View full text Add to dashboard Cite

The use of multiple senses in interactive applications has become increasingly feasible due to the upsurge of commercial, off-the-shelf devices to produce sensory effects. Creating Multiple Sensorial Media (MulSeMedia) immersive systems requires understanding their digital ecosystem. Mulsemedia systems encompass a set of applications, and devices of different types assembled to communicate or express feelings from the virtual world to the real world. Despite existing standards, tools, and recent research devoted to them, there is still a lack of formal and explicit representation of what mulsemedia is. Misconceptions could eventually lead to the construction of solutions that might not take into account reuse, integration, standardization, among other design features. In this paper, we propose to establish a common conceptualization about mulsemedia systems through a reference ontology, named MulseOnto, covering their main notions. To evaluate it, we applied ontology verification and validation techniques, including assessment by humans and a data-driven approach. The results showed that MulseOnto can be used as a consensual conceptual model for exploring the knowledge about the whole chain of mulsemedia systems. CCS CONCEPTS • General and reference → Reference works; • Information systems → Multimedia information systems; • Human-centered computing → Ubiquitous and mobile computing theory, concepts and paradigms;

show abstract

“…Part 3: "Sensory Information" of the standard [ISO/IEC 2013] defines a set of sensory effects (e.g., light, temperature, wind, vibration, touch) as well as semantics that the content creator may use to deliver multisensorial content in association with audiovisual data. Waltl et al [2013] and Yoon [2013] demonstrated an effective end-to-end framework implementation for the creation and delivery of multi-sensorial data synchronized with audiovisual content using the MPEG-V standard.…”

Section: Related Workmentioning

confidence: 99%

Integration of Multisensorial Stimuli and Multimodal Interaction in a Hybrid 3DTV System

Luque

Galloso

Feijóo

et al. 2014

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

This paper proposes the integration of multi-sensorial stimuli and multi-modal interaction components into a sports multimedia asset under two dimensions: immersion and interaction. The first dimension comprises a binaural audio system and a set of sensory effects synchronized with the audiovisual content, whereas the second explores interaction through the insertion of interactive 3D objects into the main screen and on-demand presentation of additional information in a second touchscreen. We present an end-to-end solution integrating these components into a hybrid (internet-broadcast) television system using current 3DTV standards. Results from an experimental study analyzing the perceived quality of these stimuli and their influence in the Quality of Experience are presented. General Terms: Multimedia Information SystemsAdditional Key Words and Phrases: multi-sensorial multi-modal media, immersive media, interactive media, hybrid-based 3DTV, binaural audio, sensory effects, interactive 3D objects integrated into the video scene, second screen, quality of experience. ACM Reference Format:Francisco Pedro Luque, Iris Galloso, Carlos Alberto Martín, Claudio Feijoo and Guillermo Cisneros. 2014. Integration of multisensorial stimuli and multi-modal interaction in a hybrid 3DTV system.

show abstract

An end-to-end tool chain for Sensory Experience based on MPEG-V

Cited by 63 publications

References 17 publications

QoE of cross-modally mapped Mulsemedia: an assessment using eye gaze and heart rate

QoE of cross-modally mapped Mulsemedia: an assessment using eye gaze and heart rate

Towards a reference ontology on mulsemedia systems

Integration of Multisensorial Stimuli and Multimodal Interaction in a Hybrid 3DTV System

Contact Info

Product

Resources

About