Time–frequency scattering accurately models auditory similarities between instrumental playing techniques

Abstract

Instrumentalplaying techniques such as vibratos, glissandos, and trills often denote musical expressivity, both in classical and folk contexts. However, most existing approaches to music similarity retrieval fail to describe timbre beyond the so-called “ordinary” technique, use instrument identity as a proxy for timbre quality, and do not allow for customization to the perceptual idiosyncrasies of a new subject. In this article, we ask 31 human participants to organize 78 isolated notes into a set of timbre clusters. Analyzing their responses suggests that timbre perception operates within a more flexible taxonomy than those provided by instruments or playing techniques alone. In addition, we propose a machine listening model to recover the cluster graph of auditory similarities across instruments, mutes, and techniques. Our model relies on joint time–frequency scattering features to extract spectrotemporal modulations as acoustic features. Furthermore, it minimizes triplet loss in the cluster graph by means of the large-margin nearest neighbor (LMNN) metric learning algorithm. Over a dataset of 9346 isolated notes, we report a state-of-the-art average precision at rank five (AP@5) of .%. An ablation study demonstrates that removing either the joint time–frequency scattering transform or the metric learning algorithm noticeably degrades performance.

Perceptual Musical Similarity Metric Learning with Graph Neural Networks

“…The Cyberlioz dataset, introduced in [3] as Cyberlioz-v1, is a collection of human similarity judgements between unique musical instrument playing techniques from 31 subjects. The dataset is based on the perceptual similarity judgements of 78 reference sounds, sourced from the comprehensive Studio-on-Line (SOL) (v0.9) dataset [11], sampled at 44.1 kHz.…”

“…A recent publication [3] sought to address these limitations by focusing on similarity retrieval in terms of human similarity judgements between musical instrument playing techniques (IPTs), or instrument-mute-technique classes (IMT), from the Studio-on-Line (SOL) dataset. By using time-frequency scattering acoustic features [4], they reported 90% (with a Euclidean metric) and 99% (with a learned linear metric) top-5 similarity retrieval accuracy.…”

Fourier at the heart of computer music: From harmonic sounds to texture

“…, y 50 iteratively, thus converging towards x in the sense of the associated Euclidean distance functional E x , with Q = 12 and T = 188 ms. The resulting piece, named FAVN, was premiered at the Alte Oper in Frankfurt, presented again at the Geometry of Now festival in Moscow, and became a two-month exhibition at the Kunsthalle in Vienna, with a dedicated retrospective catalogue [28]. At the time of the concert, the performer of FAVN has to play back the first iteration of the first fragment, and then move forward progressively in the digital reproduction of the piece, both in terms of compositional time (fragments) and computational time (iterations).…”

“…Beyond the aesthetic of experimental music, time-frequency scattering has recently found a wider audience by appearing on an electronica record named The Shape of RemiXXXes to Come, released by the independent record label Warp [29]. In his remix, Hecker isolated a few one-bar loops from Lorenzo Senni's XAllegroX and reconstructing them from their time-frequency scattering coefficients.…”

Deep scattering network for speech emotion recognition

“…Along similar lines, [18] also applies to scatter coefficients for environmental sound classification. In [19] a joint time-frequency based scattering representation is used to analyze the timbral similarity between different acoustic instruments. Our work is the first to apply scattering transform for SER.…”