The ecology of human language is face-to-face interaction, comprising cues such as prosody, co-speech gestures and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants while recording their electroencephalogram. We quantified multimodal cues (prosody, gestures, mouth movements) and measured their effect on a well-established electroencephalographic marker of processing load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, they were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.

Onomatopoeia is widespread across the world’s languages. They represent a relatively simple iconic mapping: the phonological/phonetic properties of the word evokes acoustic related features of referents. Here, we explore the EEG correlates of processing onomatopoeia in English. Participants were presented with a written cue-word (e.g., leash) and then with a spoken target-word. The target-word was either an onomatopoeia (e.g., bark), a sound-related but arbitrary word (e.g., melody), or another arbitrary word (e.g., bike). Participants judged whether the cue- and the target-word were similar in meaning. We analysed Event-Related Potentials (ERPs) in different time-windows: (i) early (100–200 and 200–250 ms) to assess differences in processing at the form-level; (ii) the N400 time-window (300–500 ms) in order to establish if there are differences in semantic processing across our word-types; and (iii) late (600–900 ms) to assess post-lexical effects. We found that onomatopoeia differed from the other words in the N400 time-window: when cue and target were unrelated, onomatopoeic words led to greater negativity which can be accounted for in terms of enhanced semantic activation of onomatopoeia which leads to greater salience of the mismatch. We discuss results in the context of a growing body of literature investigating iconicity in language processing and development.

The natural ecology of human language is face-to-face interaction, comprising cues, like cospeech gestures, mouth movements and prosody, tightly synchronized with speech. Yet, this rich multimodal context is usually stripped away in experimental studies as the dominant paradigm focuses on speech alone. We ask how these audio-visual cues impact brain activity during naturalistic language comprehension, how they are dynamically orchestrated and whether they are organized hierarchically. We quantify each cue in video-clips of a speaker and we used a well-established electroencephalographic marker of comprehension difficulties, an event-related potential, peaking around 400ms after word-onset. We found that multimodal cues always modulated brain activity in interaction with speech, that their impact dynamically changes with their informativeness and that there is a hierarchy: prosody shows the strongest effect followed by gestures and mouth movements. Thus, this study provides a first snapshot into how the brain dynamically weights audiovisual cues in real-world language comprehension. Electrophysiology of multimodal comprehension ! 3 Electrophysiology of multimodal comprehension ! 4 frame theories of natural language processing because if some multimodal cues (e.g., gesture or prosody) always contribute to processing, this would imply that our current speech-only focus is too narrow, if not misleading. Second, we need to understand the dynamics of online multimodal comprehension. In particular, to provide mechanistic accounts of language comprehension, it is necessary to establish how the weight of a certain cue dynamically changes depending upon the context (e.g., whether meaningful hand gestures are weighted more when prior linguistic context is less informative and/or when mouth movements are less informative). Finally, it is important to establish whether there is a stable hierarchical organization of cues (e.g., prior linguistic context may always be weighted more than gestures, which are in turn weighted more than mouth movements). Prosody, gesture and mouth movements as predictors of upcoming words: the state of the artAccentuation (i.e., prosodic stress characterized as higher pitch that makes words acoustically prominent) marks new information 10 . Many behavioural studies have revealed that comprehension is facilitated with appropriate accentuation (new information is accentuated, and old information de-accentuated 11,12 . Incongruence between the presence of prosodic accentuation and newness of information increases processing difficulty, inducing increased activation in left inferior frontal gyrus, interpreted as increased phonological and semantic processing difficulty 13 . In electrophysiological (EEG) studies, such mismatch elicits more negative N400 (an event-related-potential (ERP) peaking negatively 400ms after word presentation around central-parietal areas 14 , that has been argued to mark prediction in language comprehension 2 ) than appropriate accentuation 15-20 .Electrophysiology of multimod...