The Sounds of Sentences: Differentiating the Influence of Physical Sound, Sound Imagery, and Linguistically Implied Sounds on Physical Sound Processing

Abstract

Both the imagery literature and grounded models of language comprehension emphasize the tight coupling of high-level cognitive processes, such as forming a mental image of something or language understanding, and low-level sensorimotor processes in the brain. In an electrophysiological study, imagery and language processes were directly compared and the sensory associations of processing linguistically implied sounds or imagined sounds were investigated. Participants read sentences describing auditory events (e.g., BThe dog barks^), heard a physical (environmental) sound, or had to imagine such a sound. We examined the influence of the 3 sound conditions (linguistic, physical, imagery) on subsequent physical sound processing. Event-related potential (ERP) difference waveforms indicated that in all 3 conditions, prime compatibility influenced physical sound processing. The earliest compatibility effect was observed in the physical condition, starting in the 80-110 ms time interval with a negative maximum over occipital electrode sites. In contrast, the linguistic and the imagery condition elicited compatibility effects starting in the 180-220 ms time window with a maximum over central electrode sites. In line with the ERPs, the analysis of the oscillatory activity showed that compatibility influenced early theta and alpha band power changes in the physical, but not in the linguistic and imagery, condition. These dissociations were further confirmed by dipole localization results showing a clear separation between the source of the compatibility effect in the physical sound condition (superior temporal area) and the source of the compatibility effect triggered by the linguistically implied sounds or the imagined sounds (inferior temporal area). Implications for grounded models of language understanding are discussed.

Environmental sound priming: Does negation modify N400 cross-modal priming effects?

“…If the cross-modal priming effect is modulated by sentential meaning differences, we expect the N400 match-mismatch effect to be modified by the negation operator. In contrast, if language-sound priming takes place on the word-level, we again expect to observe the word-sound priming effect as previously reported on the N400 (e.g., Dudschig et al, 2016a), but no modification of this N400 priming effect by the negation operator.…”

“…Electroencephalographic (EEG) activity was sampled at 512 Hz from 72 Ag/AgCl electrodes. EEG/ERP analysis was performed using MATLAB toolboxes (Delorme & Makeig, 2004;Oostenveld et al, 2011) and custom MATLAB scripts (see Dudschig et al, 2016a). The analysis epoch started 1 s prior to sound onset lasting 2.5 s.…”

“…We used the original 80 German sentences from Dudschig et al (2016a) in an affirmative and a negated version [e.g., Die Katze miaut jetzt/nicht (the cat meows now/not), Die Tür quietscht jetzt/nicht (The door squeals now/not)], resulting in a total of 160 sentences. We also used the original 80 sounds and a conceptually identical set of 80 sounds in order to avoid target repetitions (sampled rate 44.1 kHz, 16-bit mono wav files, 500 ms length; see also Shafiro & Gygi, 2004).…”

“…Despite the use of sentential materials, it remains an open issue whether the observed priming effects occur on a sentential level or are rather driven by single words, since this study did not control for possible influences from single words (e.g., the single word 'waves' might already prime the sound of waves). A similar limitation applies to a recent study in which we investigated whether effects from sentences on sound processing are comparable to sensory priming between physical sounds-as suggested by the mental simulation account-or are mediated on a conceptual level (Dudschig, Mackenzie, Strozyk, Kaup, & Leuthold, 2016a). In that study, we used simple sentences (e.g., 'The dog barks') and investigated the effects on the processing of subsequent environmental sounds.…”

“…Interestingly, to our knowledge, to date, no study has directly investigated whether cross-model association effects are influenced by linguistic negation. Here we use the sentences from Dudschig et al (2016a) but this time in an affirmative [e.g., Der Hund bellt jetzt (the dog barks now)] and negated version [e.g., Der Hund bellt nicht (the dog does not bark)]. If the cross-modal priming effect is modulated by sentential meaning differences, we expect the N400 match-mismatch effect to be modified by the negation operator.…”