Exploring the role of low level visual processing in letter-speech sound integration: a visual MMN study

Abstract

In contrast with for example audiovisual speech, the relation between visual and auditory properties of letters and speech sounds is artificial and learned only by explicit instruction. The arbitrariness of the audiovisual link together with the widespread usage of letter–speech sound pairs in alphabetic languages makes those audiovisual objects a unique subject for crossmodal research. Brain imaging evidence has indicated that heteromodal areas in superior temporal, as well as modality-specific auditory cortex are involved in letter–speech sound processing. The role of low level visual areas, however, remains unclear. In this study the visual counterpart of the auditory mismatch negativity (MMN) is used to investigate the influences of speech sounds on letter processing. Letter and non-letter deviants were infrequently presented in a train of standard letters, either in isolation or simultaneously with speech sounds. Although previous findings showed that letters systematically modulate speech sound processing (reflected by auditory MMN amplitude modulation), the reverse does not seem to hold: our results did not show evidence for an automatic influence of speech sounds on letter processing (no visual MMN amplitude modulation). This apparent asymmetric recruitment of low level sensory cortices during letter–speech sound processing, contrasts with the symmetric involvement of these cortices in audiovisual speech processing, and is possibly due to the arbitrary nature of the link between letters and speech sounds.

Multi-sensory learning and learning to read

“…If the incongruent deviant letter "o" would evoke an auditory MMN, despite the standard speech sound /a/ being unaltered, then this effect would be comparable to the McGurk effect and indicate that the mechanism for letter-speech sound integration is comparable to audiovisual speech integration. However, the deviant letter did not evoke an auditory MMN (Froyen et al, 2010) pointing to an integration mechanism for letter-speech sound pairs that is different from that for audiovisual speech. A further argument for this difference may be inferred from the fact that all studies in which an aMMN was evoked by a deviating visual part of an audiovisual stimulus employed an auditory illusion (Besle et al, 2005).…”

“…In analogy to the use of the auditory MMN-paradigm to show influences of letters on speech sound processing, we now used the visual counterpart of the MMN (vMMN) in a cross-modal design to investigate the influences of speech sounds on letter processing (Froyen et al, 2010). The vMMN is described as a negativity measured at the occipital electrodes between 150 and 350 ms after the onset of an infrequent (deviant) visual stimulus in a sequence of frequently presented (standard) visual stimuli (Czigler, 2007;Pazo-Alvarez et al, 2003).…”

Congruency of auditory sounds and visual letters modulates mismatch negativity and P300 event-related potentials

“…Likewise, counterbalancing presentation of the key auditory and visual vowel stimuli meant that condition-wise differences cannot be due to vagaries in how the perceptual system processes the specific vowels or letters. A recent study examined the effect of AV congruency during a visual analogue to the MMN paradigm (Froyen et al, 2010). Subjects viewed a stream of repeated letters that was interspersed with occasional oddball items.…”

The neural signature of orthographic–phonological binding in successful and failing reading development

“…Second, the posterior network for integrating letters and speech sounds not only uses temporal proximity, but also content to determine if letter-speech sound pairs constitute meaningful associations. One issue remains open however; all the above studies were conducted with fully transparent letter-sound pairs; the used letters were always pronounced the same in each of the involved orthographies, respectively Finnish, Japanese Kana , 2007aFroyen et al, 2008Froyen et al, , 2010Blau et al, 2009. if letter-sound relations are processed in a similar fashion in an extremely opaque orthography like English, in which the relations between letters and speech sounds are deeply ambiguous (Share, 2008) and children often initially learn to associate letter names, and not speech sounds, with letters (e.g., Evans et al, 2006).…”