Disentangling stimulus plausibility and contextual congruency: Electro-physiological evidence for differential cognitive dynamics

Abstract

A B S T R A C TExpectancy mechanisms are routinely used by the cognitive system in stimulus processing and in anticipation of appropriate responses. Electrophysiology research has documented negative shifts of brain activity when expectancies are violated within a local stimulus context (e.g., reading an implausible word in a sentence) or more globally between consecutive stimuli (e.g., a narrative of images with an incongruent end). In this EEG study, we examine the interaction between expectancies operating at the level of stimulus plausibility and at more global level of contextual congruency to provide evidence for, or against, a disassociation of the underlying processing mechanisms. We asked participants to verify the congruency of pairs of cross-modal stimuli (a sentence and a scene), which varied in plausibility. ANOVAs on ERP amplitudes in selected windows of interest show that congruency violation has longer-lasting (from 100 to 500 ms) and more widespread effects than plausibility violation (from 200 to 400 ms). We also observed critical interactions between these factors, whereby incongruent and implausible pairs elicited stronger negative shifts than their congruent counterpart, both early on (100-200 ms) and between 400-500 ms. Our results suggest that the integration mechanisms are sensitive to both global and local effects of expectancy in a modality independent manner. Overall, we provide novel insights into the interdependence of expectancy during meaning integration of cross-modal stimuli in a verification task.

No language unification without neural feedback: How awareness affects sentence processing

“…In EEG, incongruent word(s)-picture pairs triggered an N400 effect in both unmasked sentence and single word trials. This is in line with previous work, showing that manipulating semantic congruency within word-picture pairs elicits N400 effects (Coco et al, 2017;Federmeier and Kutas, 2001;Willems et al, 2008). On the contrary, in the masked condition, an N400 effect was only observed for single words and not for sentences.…”

Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension

“…Indeed, while we here focused on wordless visual narratives, image sequences typically appear in multimodal relationships, such as combined with written language. ERP research has shown that visual narratives integrated with text (Manfredi et al., ) and paired with auditory speech/sounds (Manfredi, Cohn, De Araújo Andreoli, & Boggio, ) elicit N400s and late effects with the same time course as in unimodal contexts, consistent with other studies of crossmodal and multimodal processing (Coco, Araujo, & Petersson, ; Liu, Wang, & Jin, ; Liu, Wang, Wu, & Meng, ; Weissman & Tanner, ; Wu & Coulson, ). Such findings support that the N400 reflects semantic activation that is only semisensitive to modality‐specific inputs (as suggested by variance in the scalp distribution of the N400 to different modalities).…”

Fixation-related brain activity during semantic integration of object-scene information

“…The corresponding FRP waveforms showed a negative shift over frontal and central scalp sites when participants fixated a scene-inconsistent object. This result is in agreement with traditional ERP studies that have shown an frontocentral N300/N400 complex after passive foveal stimulation (e.g., Coco et al, 2017;Ganis & Kutas, 2003;Mudrik et al, 2014;Võ & Wolfe, 2013) and extends this finding for the first time to a natural viewing situation with eye-movements. Importantly, the time course of this effect suggests that it was already initiated during the preceding fixation (t-1), but then carried on through fixation (t) on the target object.…”

“…This long-lasting negative shift typically starts as early as 200-250 ms after stimulus onset (Mudrik, Shalgi, Lamy, & Deouell, 2014;Draschkow, Heikel, Võ, Fiebach, & Sassenhagen, 2018) and has its maximum at frontocentral scalp sites, in contrast to the centroparietal N400 effect for words (e.g., Kutas & Federmeier, 2011). The effect was found when the object appeared at a cued location after the scene background was already shown (Ganis & Kutas, 2003), for objects that were photoshopped into the scene (Mudrik, Lamy, & Deouell, 2010;Mudrik, et al, 2014;Coco, Araujo, & Petersson, 2017), and for objects that were part of realistic photographs (Võ & Wolfe, 2013). ERP effects of object-scene consistency have typically been subdivided into two distinct components: N300 and N400.…”