Agreement among human and automated estimates of similarity in a global music sample

Abstract

While music information retrieval (MIR) has made substantial progress in automatic analysis of audio similarity for Western music, it remains unclear whether these algorithms can be meaningfully applied to cross-cultural analyses of more diverse samples. Here we collected perceptual ratings from 62 participants using a global sample of 30 traditional songs, and compared these ratings against both pre-existing expert annotations and state-of-the-art audio similarity algorithms. We found that different methods of perceptual ratings all produced similar, moderate levels of inter-rater reliability comparable to previous studies, but that agreement between human and automated methods was always low regardless of the specific methods used to calculate musical similarity. Our findings suggest that current MIR methods are unable to measure cross-cultural music similarity in perceptually meaningful ways. We propose future directions to enable meaningful automatic analysis of all the world’s music.

Cross-cultural perception of musical similarity within and between India and Japan

“…We chose the same 30 recordings used by Daikoku et al (2022) in order to compare perceptual responses between Japanese and Indian raters. The 30 recordings consist of a collection of traditional songs from Alan Lomax's Cantometrics Consensus tapes (Lomax, 1976;Wood, 2020).…”

“…These recordings were originally designed to test human raters after being trained to use Lomax's 37-feature Cantometric rating system (Lomax, 1968;Savage, 2018;Wood, 2020;Wood et al, 2021). We used the 10s excerpts of these 30 recordings from Daikoku et al (2022) to be able to directly compare results (10s excerpts were originally chosen to ensure the experiments could be conducted within a feasible time). Daikoku et al, 2022) showing the approximate locations and cultural group names of the 30-song sample from the Cantometrics Consensus Tape.…”

“…We used the 10s excerpts of these 30 recordings from Daikoku et al (2022) to be able to directly compare results (10s excerpts were originally chosen to ensure the experiments could be conducted within a feasible time). Daikoku et al, 2022) showing the approximate locations and cultural group names of the 30-song sample from the Cantometrics Consensus Tape. Songs are labeled by solo vs. group singing and with vs. without instrumental accompaniment.…”

“…However, like other aspects of music cognition, empirical data about music similarity has been largely restricted to perception of similarity in Western music by Western listeners (Jacoby et al, 2020). One of the few exceptions investigated perceptual similarity ratings by Japanese participants in a global sample of 30 songs (Daikoku et al, 2022). They found weak but significant agreement among human raters and between these human ratings and estimates of similarity calculated from expert Cantometric classifications.…”

“…In this study we ask 48 raters from India to rate musical similarity from a global sample of 30 songs chosen from Alan Lomax's Cantometrics Training Tapes (Lomax, 1976;Wood, 2020). We compare these results with data previously collected on the same set of songs from 62 Japanese participants (Daikoku et al, 2022) to measure the degree of agreement in similarity perception within and between cultures. We pre-registered our experimental design and our hypothesis that average similarity ratings among Japanese and Indian raters would be correlated, due to a combination of universals of music perception and shared experience listening to globalized music ([authors redacted], 2022).…”