Audio-visuomotor processing in the Musician's brain: an ERP study on professional violinists and clarinetists

Abstract

The temporal dynamics of brain activation during visual and auditory perception of congruent vs. incongruent musical video clips was investigated in 12 musicians from the Milan Conservatory of music and 12 controls. 368 videos of a clarinetist and a violinist playing the same score with their instruments were presented. The sounds were similar in pitch, intensity, rhythm and duration. To produce an audiovisual discrepancy, in half of the trials, the visual information was incongruent with the soundtrack in pitch. ERPs were recorded from 128 sites. Only in musicians for their own instruments was a N400-like negative deflection elicited due to the incongruent audiovisual information. SwLORETA applied to the N400 response identified the areas mediating multimodal motor processing: the prefrontal cortex, the right superior and middle temporal gyrus, the premotor cortex, the inferior frontal and inferior parietal areas, the EBA, somatosensory cortex, cerebellum and SMA. The data indicate the existence of audiomotor mirror neurons responding to incongruent visual and auditory information, thus suggesting that they may encode multimodal representations of musical gestures and sounds. These systems may underlie the ability to learn how to play a musical instrument.

When a hit sounds like a kiss: An electrophysiological exploration of semantic processing in visual narrative

“…Similar N400 effects have also been observed when information is presented across different sensory modalities, for example, during the combination of speech and/or natural sounds with inconsistent pictures or video frames (Plante et al, 2000; Puce et al 2007; Cummings et al, 2008; Liu, 2011), gestures with inconsistent verbal information (Wu and Coulson, 2005; Wu and Coulson, 2007a; Wu and Coulson, 2007b; Cornejo, 2009; Proverbio, 2014a), gestures and music (Proverbio et al, 2014b). Others have investigated cross-modal semantic processing by substituting an element from one modality for an element in another modality: e.g., substitution of a picture of an object in place of a word in sentences to examine the extent to which pictures and words may access a common semantic system (Ganis et al, 1996; Nigam et al, 1992).…”

“…Similar results have been obtained for gestures combined with verbal information, where the (in)congruity of information across the two modalities modulates the amplitude of N400 effects (Wu and Coulson, 2005; Wu and Coulson, 2007a; Wu and Coulson, 2007b; Cornejo, 2009; Proverbio, 2014a). The congruity of gesture-music pairings also affects N400 amplitudes, at least in musicians (Proverbio et al, 2014b). Although present, masked priming paradigms show a much weaker and later N400-like effect in cross-modal repetition priming (verbal vs. visual) than within-modality repetition priming (Holcomb and Anderson, 1993; Holcomb et al, 2005; Sitnikova et al, 2008).…”

Enhancement of numeric cognition in children with low achievement in mathematic after a non-instrumental musical training

“…On the other hand, Non-instrumental musical training (NIMT) is reasoned as a multimodal training with an explicit methodology since it comprises teaching strategies to recognize musical properties, making children conscious of the material, not just repeating a sequence of auditory/motor movements on a particular instrument, such as the piano (Klingberg, 2010;Ribeiro & Santos, 2015). As an outcome, NIMT promotes multi-sensory brain stimulation (Proverbio, Calbi, Manfredi, & Zani, 2014), related to visual, auditory, and sensorimotor modalities causing an improvement of audio-visual integration due to the musical activities .…”

Audio–visual integration through the parallel visual pathways

“…Electrophysiology and fMRI investigation showed that the insula is participating in multimodal integration (Calvert et al, 2001) and recently it was demonstrated that isochronous timing in an audiovisual task has a positive influence on insular activity (Marchant and Driver, 2013). Also several researches pointed out the importance of cerebellum in audio-visual integration (Pfordresher et al, 2014;Proverbio et al, 2014).…”

“…It has also been shown that a greater BOLD response in the visual cortex correlates with a higher FA in the optic radiation (Toosy et al, 2004). Neuroimaging and histological studies have been combined recently that demonstrated that spatial learning and motor learning result in locally enhanced myelination (BlumenfeldKatzir et al, 2011;Sampaio-Baptista et al, 2013), which is reflected in enhanced FA in the diffusion MRI.…”