SOUND OF(F): Contextual Storytelling Using Machine Learning Representations of Sound and Music

The Role of Audio in Immersive Storytelling: a Systematic Review in Cultural Heritage

“…An asymmetric co-occurrence matrix is created by calculating the concepts' relative co-occurrence frequencies. This matrix produces a two-dimensional concept map using a proprietary clustering algorithm based on Audio First [152][153][154][155][156] Audio and Multimodal Integration [157][158][159][160][161][162][163][164][165][166] Audio in Multimodal Comparison [167] Multimodal Experience [168][169][170][171][172][173][174][175] the spring-force model for the many-body problem [83]. The correctness of each concept in this semantic network generates a third hierarchical dimension, which displays the more general parent concepts at the higher levels.…”

“…23 studies use tangible devices, such as console games or VR headset controllers [86,128,133,144,145,149,152,159,165,168,175,186], haptic interfaces [158] or custom-built devices. In ec(h)o [61], users indicate a particular preference for presented audio content by rotating a wooden cube with coloured faces whose position is detected by a camera.…”

“…Soundscape User model detected by user position and orientation in space [121] User position and/or head orientation [56] Interaction with the virtual environment and AI techniques [165] Virtual Environment and Story User interaction with tangible interface [86,158,160] Interaction with voice [86,160] Directions User position and/or head orientation [79,128,132,135,136] Storyline Interaction with HMD headset controller [108,109,116,128,159,164,185] AI content generation [108,109] Interaction with a tangible device [61] Content created by users User interaction with tangible interface [152] Collaborative experience [161,171] especially in tourist or educational experiences, sound effects are used to create soundscapes and personalized content. In Ec(h)o [61], audio soundscape and content are based on user preferences selected by choosing the orientation of a physical cube.…”