Fusion confusion

Abstract

Virtual Reality (VR) is attracting the attention of application developers for purposes beyond entertainment including serious games, health, education and training. By including 3D audio the overall VR quality of experience (QoE) will be enhanced through greater immersion. Better understanding the perception of spatial audio localisation in audio-visual immersion is needed especially in streaming applications where bandwidth is limited and compression is required. This paper explores the impact of audio-visual fusion on speech due to mismatches in a perceived talker location and the corresponding sound using a phenomenon known as the McGurk effect and binaurally rendered Ambisonic spatial audio. The illusion of the McGurk effect happens when a sound of a syllable paired with a video of a second syllable, gives the perception of a third syllable. For instance the sound of /ba/ dubbed in video of /ga/ will lead to the illusion of hearing /da/. Several studies investigated factors involved in the McGurk effect, but a little has been done to understand the audio spatial effect on this illusion. 3D spatial audio generated with Ambisonics has been shown to provide satisfactory QoE with respect to localisation of sound sources which makes it suitable for VR applications but not for audio visual talker scenarios.In order to test the perception of the McGurk effect at different direction of arrival (DOA) of sound, we rendered Ambisonics signals at the azimuth of 0°, 30°, 60°, and 90°to both the left and right of the video source. The results show that the audio visual fusion significantly affects the perception of the speech. Yet the spatial audio does not significantly impact the illusion. This finding suggests that precise localisation of speech audio might not be as critical for speech intelligibility. It was found that a more significant factor was the intelligibility of speech itself.CCS Concepts: • Human-centered computing → Virtual reality.

The McGurk Effect Across Languages

“…Another limitation is the use of monophonic or stereophonic playback (using one or two audio channels) through speakers or headphones, which makes it difficult to mimic true ambient sound as it is easier to localize the origin of the signal, whereas in a noisy environment, the disruptive signals that confuse a listener can be projected from any direction (Snow, 1953;Yao et al, 2020). This is addressed in another study, where a McGurk-style experiment was realized using ambisonic playback, meaning audio was played back through multiple channels to simulate sounds coming from all directions (Siddig, Ragano, Jahromi & Hines, 2019); however, like much of the work on the McGurk effect, the ambisonic study was done in English. Thus, future research could explore the McGurk effect using ambisonic sound in more phonologically and tonally diverse languages.…”

InterviewBot: Real-Time End-to-End Dialogue System for Interviewing Students for College Admission

“…Although these dialogue systems have shown their effectiveness in achieving their goals, they all heavily rely on design templates. Conversational agents for interviews have been experimented with for law enforcement [25], healthcare [26], job application [27], and psychology [28], among which most are proof of concept. A few interview bots have been developed on commercial platforms such as Google Dialogflow and IBM Watson Assistant, with the limitation of pre-scripted interviews; thus, they cannot proactively follow up on the user content.…”