Multisensory Integration Is Modulated by Auditory Sound Frequency and Visual Spatial Frequency

Abstract

Accurate integration of auditory and visual information is essential for our ability to communicate with others. Previous studies have shown that the temporal discrepancies over which audiovisual speech stimuli will be integrated into a coherent percept are much wider than those typically observed for simple stimuli like beeps and flashes of light. However, our sensitivity to the low-level features of simple stimuli is not constant. We hypothesized that part of the enhanced integration of audiovisual speech may be due to it consisting predominantly of the sound frequencies and visual spatial frequencies that humans are most sensitive to. Here, we examined integration behaviors for pure tones across the sound frequency spectrum and visual gratings across the spatial frequency spectrum to examine how these low-level features modulate integration. The temporal window of integration was modulated by both sound frequency and visual spatial frequency, with the widest integration window occurring when both stimuli fell within their respective peak sensitivity ranges. These results suggest that part of the increased tolerance for temporal asynchrony typically observed for audiovisual speech may be due to the differential integration of low-level stimulus features that are dominant within complex audiovisual speech.

Perceived Congruency in Audiovisual Stimuli Consisting of Gabor Patches and AM and FM Tones

“…The congruent pairs demonstrated significantly shorter reaction times in a speeded classification task. Green et al (2019) also demonstrated a crossmodal correspondence between a spatial frequency of a Gabor patch and the frequency of a tone. However, results…”

“…Gabor patch and a sound also hold for longer stimuli remains unknown. Besides, the congruency of the stimuli was most often studied by assessing congruency effect, i.e., analysing the changes in response time or temporal window of integration in speed classification or search tasks (Evans and Treisman, 2010;Guzman-Martinez et al, 2012;Orchard-Mills et al, 2013a;Green et al, 2019). However, it is unknown whether stimuli examined in these studies are actually perceived as congruent or incongruent by participants, as no (in)congruency judgement was explicitly asked from participants.…”

“…The main objective of this dissertation is to define the parameters that affect the perceived congruency of Gabor patches and modulated tones, i.e, ( 1) to compare the perceived congruency evoked by AM-and FM-tones in stimuli consisting of a Gabor patch and a modulated tone; (2) to confirm whether static parameters have an effect on perceived congruency in the presence of dynamic changes in relatively long audiovisual stimuli (2 seconds), as it was demonstrated for short stimuli with a static Gabor patch (Evans and Treisman, 2010;Green et al, 2019); and (3) to assess the effect of correspondence between the physical frequencies of the auditory and the visual components of the stimuli on perceived audiovisual congruency.…”

“…Evans and Treisman (2010), for example, studied short stimuli (120 ms) and demonstrated crossmodal correspondence between the spatial frequency of the patch and perceived pitch of the tone. Green et al (2019) also used short (50 ms) stimuli and showed multisensory integration based on correspondence between spatial frequency of Gabor patches and sound frequency (pure tones). As the current study did not observe any evidence suggesting that perceived congruency of audiovisual stimuli might be significantly affected by such correspondence, it might be assumed, that the role of static parameters (i.e., spatial frequency of the patch and carrier frequency of the tone) is different in stimuli of different durations.…”

“…Evans and Treisman (2010) demonstrated crossmodal correspondence between the spatial frequency of a Gabor patch and a perceived pitch of a tone. Following this research, various studies used the principal of combining high (low) spatial frequency Gabor patches with high (low) tones to create (assumingly) congruent audiovisual stimuli and high (low) spatial frequency Gabor patches with low (high) tones to create (assumingly) incongruent audiovisual stimuli (Heron et al, 2012;Altieri et al, 2015;Green et al, 2019). However, the results of these studies are ambiguous.…”