Auditory, Visual and Audiovisual Speech Processing Streams in Superior Temporal Sulcus

Abstract

The human superior temporal sulcus (STS) is responsive to visual and auditory information, including sounds and facial cues during speech recognition. We investigated the functional organization of STS with respect to modality-specific and multimodal speech representations. Twenty younger adult participants were instructed to perform an oddball detection task and were presented with auditory, visual, and audiovisual speech stimuli, as well as auditory and visual nonspeech control stimuli in a block fMRI design. Consistent with a hypothesized anterior-posterior processing gradient in STS, auditory, visual and audiovisual stimuli produced the largest BOLD effects in anterior, posterior and middle STS (mSTS), respectively, based on whole-brain, linear mixed effects and principal component analyses. Notably, the mSTS exhibited preferential responses to multisensory stimulation, as well as speech compared to nonspeech. Within the mid-posterior and mSTS regions, response preferences changed gradually from visual, to multisensory, to auditory moving posterior to anterior. Post hoc analysis of visual regions in the posterior STS revealed that a single subregion bordering the mSTS was insensitive to differences in low-level motion kinematics yet distinguished between visual speech and nonspeech based on multi-voxel activation patterns. These results suggest that auditory and visual speech representations are elaborated gradually within anterior and posterior processing streams, respectively, and may be integrated within the mSTS, which is sensitive to more abstract speech information within and across presentation modalities. The spatial organization of STS is consistent with processing streams that are hypothesized to synthesize perceptual speech representations from sensory signals that provide convergent information from visual and auditory modalities.

The spatio‐temporal profile of multisensory integration

“…In accord, several previous imaging studies have linked the STS to audiovisual integration (e.g. Hein & Knight, ; Stevenson & James, ; Noesselt et al ., , ; Werner & Noppeney, ; Watson et al ., ; Venezia et al ., ). Note, however, that many of these previous imaging studies have used audiovisual speech stimuli, which might not be comparable to the simple stimuli used here.…”

“…In accord, 3.39 9 10 À11 À60 À32 10 L STG (IPL: 24%) 6.86 3.45 9 10 À10 À64 À24 10 L STG (TE 3: 40%) 6.63 1 9 10 À9 À38 À22 0 L posterior Insula (Id1: 15%, Ig2: 35%) 416 7.22 6.3 9 10 À11 62 À36 10 R STG (TE 3: 7%) 6.69 7.65 9 10 À10 66 À32 18 R STG (TE 3: 12%, PF 37%) 6.69 7.67 9 10 À10 56 À38 0 R MTG (NA) 2* 3.26 7.62 9 10 À3 À14 À28 À4 L Thalamus (MGB) 30* 8.21 5.5 9 10 À13 12 À26 À6 R Thalamus ( several previous imaging studies have linked the STS to audiovisual integration (e.g. Hein & Knight, 2008;Stevenson & James, 2009;Noesselt et al, 2010Noesselt et al, , 2012Werner & Noppeney, 2010;Watson et al, 2014;Venezia et al, 2017). Note, however, that many of these previous imaging studies have used audiovisual speech stimuli, which might not be comparable to the simple stimuli used here.…”

Brain activity during traditional textbook and audiovisual‐3D learning

“…However, little is known as to how the brain processes didactic information presented in different formats. Some work has been done to characterize the differences in the evoked brain response to verbal information presented visually or aurally (e.g., Buchweitz, Mason, Tomitch, & Just, ; Venezia et al, ), and a recent study has explicitly compared brain activity during the viewing of complex scenes (movies) and reading its screenplay text (scripts) (Tikka, Kauttonen, & Hlushchuk, ). Nevertheless, there are no studies directly investigating brain activity predicting the learning achieved during both audiovisual and text lessons.…”

Neural correlates of gender congruence in audiovisual commercials for gender‐targeted products: An fMRI study

“…Research in other fields (such as the more basal level AV integration) indicate that passive viewing and listening to AV stimuli (i.e., faces and voices, faces and music, or letters and speeches) consistently elicits activation in the superior temporal gyrus (STG) (Callan et al, 2003;Erickson, Heeg, Rauschecker, & Turkeltaub, 2014;Hölig, Föcker, Best, Röder, & Büchel, 2017;Kokinous, Kotz, Tavano, & Schröger, 2015;Van Atteveldt, Formisano, Blomert, & Goebel, 2007;Venezia et al, 2017). Research in other fields (such as the more basal level AV integration) indicate that passive viewing and listening to AV stimuli (i.e., faces and voices, faces and music, or letters and speeches) consistently elicits activation in the superior temporal gyrus (STG) (Callan et al, 2003;Erickson, Heeg, Rauschecker, & Turkeltaub, 2014;Hölig, Föcker, Best, Röder, & Büchel, 2017;Kokinous, Kotz, Tavano, & Schröger, 2015;Van Atteveldt, Formisano, Blomert, & Goebel, 2007;Venezia et al, 2017).…”