Perceptual impact of gesture control of spatialization

Abstract

In two experiments, visual cues from gesture control of spatialization were found to affect auditory movement perception depending on the identifiability of auditory motion trajectories, the congruency of audiovisual stimulation, the sensory focus of attention, and the attentional process involved. Visibility of the performer's gestures improved spatial audio trajectory identification, but it shifted the listeners' attention to vision, impairing auditory motion encoding in the case of incongruent stimulation. On the other hand, selectively directing attention to audition resulted in interference from the visual cues for acoustically ambiguous trajectories. Auditory motion information was poorly preserved when dividing attention between auditory and visual movement feedback from performance gestures. An auditory focus of attention is a listener strategy that maximizes performance, due to the improvement caused by congruent visual stimulation and its robustness to interference from incongruent stimulation for acoustically unambiguous trajectories. Attentional strategy and auditory motion calibration are two aspects that need to be considered when employing gesture control of spatialization.

Perception of Surrounding Sound Source Trajectories in the Horizontal Plane: A Comparison of VBAP and Basic-Decoded HOA

“…Their results showed that increasing the number of loudspeakers improved the trajectory localization but induced a detrimental sound coloration. Finally, Marentakis and McAdams also investigated trajectory perception on a VAD but pursuing a different goal: the perceptual impact of gesture control of spatialization [31]. Their experiment compared trajectory recognition by subjects exposed to 1 -a unimodal condition (auditory stimulus alone rendered in VBAP) and 2 -a bimodal condition in which a "performer" made gestures congruent or incongruent with the sound trajectory.…”

“…The few dynamic sound localization studies we mentioned in the introduction used either a Multiple-Choice Questionnaire (MCQ) allowing allowing the subjects to choose between various fixed trajectories [31], or free 2D drawing of the trajectories [29,28]. While MCQ results are easy to analyze, this method may be too restrictive and may influence subject choices.…”