Correspondence of categorical and feature‐based representations of music in the human brain

The human linguistic system is characterized by modality invariance and attention selectivity. Previous studies have examined these properties independently and reported perisylvian region involvement for both; however, their relationship and the linguistic information they harbor remain unknown. Participants were assessed by functional magnetic resonance imaging, while spoken narratives (auditory) and written texts (visual) were presented, either separately or simultaneously. Participants were asked to attend to one stimulus when both were presented. We extracted phonemic and semantic features from these auditory and visual modalities, to train multiple, voxel-wise encoding models. Cross-modal examinations of the trained models revealed that perisylvian regions were associated with modality-invariant semantic representations. Attentional selectivity was quantified by examining the modeling performance for attended and unattended conditions. We have determined that perisylvian regions exhibited attention selectivity. Both modality invariance and attention selectivity are both prominent in models that use semantic but not phonemic features. Modality invariance was significantly correlated with attention selectivity in some brain regions; however, we also identified cortical regions associated with only modality invariance or only attention selectivity. Thus, paying selective attention to a specific sensory input modality may regulate the semantic information that is partly processed in brain networks that are shared across modalities.

show abstract

“…For the auditory regressor, we used a modulation transfer function (MTF) model ( Nakai et al. 2021 ).…”

Section: Methodsmentioning

confidence: 99%

Convergence of Modality Invariance and Attention Selectivity in the Cortical Semantic Circuit

2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…First, in our feature regression analysis, the acoustic features we selected may not represent the full range of acoustic dynamics occurring throughout each excerpt. Previous studies using encoding models to examine brain activity evoked by music employed a range of acoustic features, such as the modulation transfer function (Norman-Haignere et al, 2015;Patil et al, 2012) as well as music-related models representing mode, roughness, root mean square energy (RMS), and pulse clarity (Alluri et al, 2012;Nakai et al, 2021;Toiviainen et al, 2014). However, the types of information captured by these features are also roughly captured by the features used in this study.…”

Section: Discussionmentioning

confidence: 99%

High-order areas and auditory cortex both represent the high-level event structure of music

Williams

Margulis

Nastase

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent fMRI studies of event segmentation have found that default mode regions represent high-level event structure during movie watching. In these regions, neural patterns are relatively stable during events and shift at event boundaries. Music, like narratives, contains hierarchical event structure (e.g., sections are composed of phrases). Here, we tested the hypothesis that brain activity patterns in default mode regions reflect the high-level event structure of music. We used fMRI to record brain activity from 25 participants (male and female) as they listened to a continuous playlist of 16 musical excerpts, and additionally collected annotations for these excerpts by asking a separate group of participants to mark when meaningful changes occurred in each one. We then identified temporal boundaries between stable patterns of brain activity using a hidden Markov model and compared the location of the model boundaries to the location of the human annotations. We identified multiple brain regions with significant matches to the observer-identified boundaries, including auditory cortex, mPFC, parietal cortex, and angular gyrus. From these results, we conclude that both higher-order and sensory areas contain information relating to the high-level event structure of music. Moreover, the higher-order areas in this study overlap with areas found in previous studies of event perception in movies and audio narratives, including regions in the default mode network.Significance StatementListening to music requires the brain to track dynamics at multiple hierarchical timescales. In our study, we had fMRI participants listen to real-world music (classical and jazz pieces) and then used an unsupervised learning algorithm (a hidden Markov model) to model the high-level event structure of music within participants’ brain data. This approach revealed that default mode brain regions involved in representing the high-level event structure of narratives are also involved in representing the high-level event structure of music. These findings provide converging support for the hypothesis that these regions play a domain-general role in processing stimuli with long-timescale dependencies.

show abstract

“…We obtained 3 hours of neuroimaging data for each participant, allowing researchers to construct participant-wise modeling of brain activity. Although this dataset has been used in two previous studies [2 , 3] , there remains much room to apply different acoustic models. The original music stimuli have been widely used in previous music information retrieval studies (see [4] for a review).…”

Section: Data Descriptionmentioning

confidence: 99%

“…Test samples were measured while the sequence of test music clips was presented four times. Modified from [3] . …”

Section: Data Descriptionmentioning

confidence: 99%

Music genre neuroimaging dataset

2022

Self Cite

View full text Add to dashboard Cite

This dataset includes functional magnetic resonance imaging (fMRI) data collected while five subjects extensively listened to 540 music pieces from 10 music genres over the course of 3 days. Behavioral data are also available. Data are separated into training and test samples to facilitate the application of machine learning algorithms. Test stimuli were repeated four times and can be used to evaluate the signal to noise ratio of brain activity. Using this dataset, both neuroimaging and machine learning researchers can test multiple algorithms regarding the prediction performance of brain activity induced by various music stimuli. Although two previous studies have used this dataset, there remains much room to apply different acoustic models. This dataset can contribute to integration of the fields of auditory neuroscience and machine learning.

show abstract

Correspondence of categorical and feature‐based representations of music in the human brain

Cited by 19 publications

References 45 publications

Convergence of Modality Invariance and Attention Selectivity in the Cortical Semantic Circuit

Convergence of Modality Invariance and Attention Selectivity in the Cortical Semantic Circuit

High-order areas and auditory cortex both represent the high-level event structure of music

Music genre neuroimaging dataset

Contact Info

Product

Resources

About