Neural Basis of the Sound-Symbolic Crossmodal Correspondence Between Auditory Pseudowords and Visual Shapes

McCormick, Kelly; Lacey, Simon; Stilla, Randall; Nygaard, Lynne C.; Sathian, K.

doi:10.1101/478347

Cited by 6 publications

(10 citation statements)

References 130 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, whether performance is better or worse really does seem to depend on the task that participants have to perform. For instance, crossmodally congruent stimulus combinations typically facilitate performance on speeded discrimination ( [43]; and see [2] for a review) and Implicit Association Test-type tasks [9,44]. Meanwhile, in terms of the detrimental effects of audiovisual crossmodal correspondences, Parise and Spence [13] reported a series of unspeeded psychophysical studies demonstrating that participants found it significantly harder to discriminate the relative location, or timing, of auditory stimuli (pure tones) when presented together with a corresponding as compared to an incongruent visual stimulus.…”

Section: Correspondences Involving Simple Auditorymentioning

confidence: 99%

Simple and complex crossmodal correspondences involving audition

Spence

2020

Acoust. Sci. & Tech.

View full text Add to dashboard Cite

The last few years have seen an explosion of interest from researchers in the crossmodal correspondences, defined as the surprising connections that the majority of people share between seemingly-unrelated stimuli presented in different sensory modalities. Intriguingly, many of the crossmodal correspondences that have been documented/studied to date have involved audition as one of the corresponding modalities. In fact, auditory pitch may well be the single most commonly studied dimension in correspondences research thus far. That said, relatively separate literatures have focused on the crossmodal correspondences involving simple versus more complex auditory stimuli. In this review, I summarize the evidence in this area and consider the relative explanatory power of the various different accounts (statistical, structural, semantic, and emotional) that have been put forward to explain the correspondences. The suggestion is made that the relative contributions of the different accounts likely differs in the case of correspondences involving simple versus more complex stimuli (i.e., pure tones vs. short musical excerpts). Furthermore, the consequences of presenting corresponding versus non-corresponding stimuli likely also differ in the two cases. In particular, while crossmodal correspondences may facilitate binding (i.e., multisensory integration) in the case of simple stimuli, the combination of more complex stimuli (such as, for example, musical excerpts and paintings) may instead be processed more fluently when the component stimuli correspond. Finally, attention is drawn to the fact that the existence of a crossmodal correspondence does not in-and-ofitself necessarily imply that a crossmodal influence of one modality on the perception of stimuli in the other will also be observed.

show abstract

Section: Correspondences Involving Simple Auditorymentioning

confidence: 99%

Simple and complex crossmodal correspondences involving audition

Spence

2020

Acoust. Sci. & Tech.

View full text Add to dashboard Cite

show abstract

“…Perhaps in the presence of sound-symbolic CCs, individuals engage areas of the brain linked to the perception of speech and multisensory integration, like the posterior superior temporal sulcus (Beauchamp, Nath, & Pasalar, 2010). Alternatively, perhaps the presence of sound-symbolic CCs would differentially engage participants' frontoparietal areas that are linked to attention (McCormick et al, 2018;Peiffer-Smadja & Cohen, 2019) and abstract categorization (Revill et al, 2014). Revill et al (2014) showed that left superior parietal lobule exhibited heightened activation during presentation of sound-symbolic versus non-symbolic words, while fractional anisotropy in the superior longitudinal fasciculus, an area implicated in phonological processing and multisensory facilitation (Brang et al, 2013), correlated with performance of a sound-symbolic task.…”

Section: Discussion 41 Sound-symbolic Crossmodal Correspondences Andmentioning

confidence: 99%

“…In adults, sound symbolism may offer linguistic processing advantages for categorization and word learning, particularly for larger vocabularies (Brand et al, 2018;Gasser, 2004;Revill et al, 2018). Though most of the evidence supporting sound symbolism has consisted of behavioral studies, more recently neuroimaging studies have begun to reveal the neural correlates of this phenomenon (Revill et al, 2014;McCormick et al, 2018;Peiffer-Smadja & Cohen, 2019).…”

Section: Introductionmentioning

confidence: 99%

Acoustic and Visual Stimulus Parameters Underlying Sound Symbolic Crossmodal Correspondences

Sm¹,

McCormick²,

Lacey

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

It is often assumed that a fundamental property of language is the arbitrariness of the relationship between sound and meaning. Sound symbolism, which refers to non-arbitrary mapping between the sound of a word and its meaning, contradicts this assumption. Sensitivity to sound symbolism has been studied through crossmodal correspondences (CCs) between auditory pseudowords (e.g. 'loh-moh') and visual shapes (e.g. a blob). We used representational similarity analysis to examine the relationships between physical stimulus parameters and perceptual ratings that varied on dimensions of roundedness and pointedness, for a range of auditory pseudowords and visual shapes. We found that perceptual ratings of these stimuli relate to certain physical features of both the visual and auditory domains. Representational dissimilarity matrices (RDMs) of parameters that capture the spatial profile of the visual shapes, such as the simple matching coefficient and Jaccard distance, were significantly correlated with those of the visual ratings. RDMs of certain acoustic parameters of the pseudowords, such as the temporal fast Fourier transform (FFT) and spectral tilt, that reflect spectral composition, as well as shimmer and speech envelope that reflect aspects of amplitude variation over time, were significantly correlated with those of the auditory perceptual ratings. RDMs of the temporal FFT (acoustic) and the simple matching coefficient (visual) were significantly correlated. These findings suggest that sound-symbolic CCs are related to basic properties of auditory and visual stimuli, and thus provide insights into the fundamental nature of sound symbolism and how this might evoke specific impressions of physical meaning in natural language.

show abstract

“…We used 90 shapes, consisting of gray line drawings (RGB: 240, 240, 240) on a white background, created in Adobe Illustrator (Ventura, CA: McCormick et al, unpublished data;McCormick et al, 2018: see Fig. 1 for example) following a method similar to that of Monaghan et al (2012).…”

Section: Visual Shapesmentioning

confidence: 99%

“…In adults, sound symbolism may offer linguistic processing advantages for categorization and word learning (Brand, Monaghan, & Walker, 2018;Gasser, 2004;Revill, Namy, & Nygaard, 2018), and for rehabilitation of patients with aphasia (Meteyard, Stoppard, Snudden, Cappa, & Vigliocco, 2015). More recently, neuroimaging studies have begun to reveal the neural correlates of sound symbolism (McCormick, Lacey, Stilla, Nygaard, & Sathian, 2018;Peiffer-Smadja & Cohen, 2019;Revill et al, 2014).…”

Section: Introductionmentioning

confidence: 99%

Stimulus Parameters Underlying Sound‐Symbolic Mapping of Auditory Pseudowords to Visual Shapes

Lacey¹,

Jamal

List³

et al. 2020

Cognitive Science

Self Cite

View full text Add to dashboard Cite

Sound symbolism refers to non‐arbitrary mappings between the sounds of words and their meanings and is often studied by pairing auditory pseudowords such as “maluma” and “takete” with rounded and pointed visual shapes, respectively. However, it is unclear what auditory properties of pseudowords contribute to their perception as rounded or pointed. Here, we compared perceptual ratings of the roundedness/pointedness of large sets of pseudowords and shapes to their acoustic and visual properties using a novel application of representational similarity analysis (RSA). Representational dissimilarity matrices (RDMs) of the auditory and visual ratings of roundedness/pointedness were significantly correlated crossmodally. The auditory perceptual RDM correlated significantly with RDMs of spectral tilt, the temporal fast Fourier transform (FFT), and the speech envelope. Conventional correlational analyses showed that ratings of pseudowords transitioned from rounded to pointed as vocal roughness (as measured by the harmonics‐to‐noise ratio, pulse number, fraction of unvoiced frames, mean autocorrelation, shimmer, and jitter) increased. The visual perceptual RDM correlated significantly with RDMs of global indices of visual shape (the simple matching coefficient, image silhouette, image outlines, and Jaccard distance). Crossmodally, the RDMs of the auditory spectral parameters correlated weakly but significantly with those of the global indices of visual shape. Our work establishes the utility of RSA for analysis of large stimulus sets and offers novel insights into the stimulus parameters underlying sound symbolism, showing that sound‐to‐shape mapping is driven by acoustic properties of pseudowords and suggesting audiovisual cross‐modal correspondence as a basis for language users' sensitivity to this type of sound symbolism.

show abstract

Neural Basis of the Sound-Symbolic Crossmodal Correspondence Between Auditory Pseudowords and Visual Shapes

Cited by 6 publications

References 130 publications

Simple and complex crossmodal correspondences involving audition

Simple and complex crossmodal correspondences involving audition

Acoustic and Visual Stimulus Parameters Underlying Sound Symbolic Crossmodal Correspondences

Stimulus Parameters Underlying Sound‐Symbolic Mapping of Auditory Pseudowords to Visual Shapes

Contact Info

Product

Resources

About