Modeling the onset advantage in musical instrument recognition

Siedenburg, Kai; Schädler, Marc René; Hülsmeier, David

doi:10.1121/1.5141369

Cited by 6 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These findings are in line with a previous publication [18] which modeled perceived dissimilarity between musical sounds by means of STRF features. Likewise, a recent publication has observed similar performance in instrument identification of a human and machine classifier using separable Gabor filterbank (GBFB) features [80].…”

Section: Role Of Dataset Sizementioning

confidence: 76%

Time–frequency scattering accurately models auditory similarities between instrumental playing techniques

Lostanlen

El-Hajj

Rossignol³

et al. 2021

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Instrumentalplaying techniques such as vibratos, glissandos, and trills often denote musical expressivity, both in classical and folk contexts. However, most existing approaches to music similarity retrieval fail to describe timbre beyond the so-called “ordinary” technique, use instrument identity as a proxy for timbre quality, and do not allow for customization to the perceptual idiosyncrasies of a new subject. In this article, we ask 31 human participants to organize 78 isolated notes into a set of timbre clusters. Analyzing their responses suggests that timbre perception operates within a more flexible taxonomy than those provided by instruments or playing techniques alone. In addition, we propose a machine listening model to recover the cluster graph of auditory similarities across instruments, mutes, and techniques. Our model relies on joint time–frequency scattering features to extract spectrotemporal modulations as acoustic features. Furthermore, it minimizes triplet loss in the cluster graph by means of the large-margin nearest neighbor (LMNN) metric learning algorithm. Over a dataset of 9346 isolated notes, we report a state-of-the-art average precision at rank five (AP@5) of .%. An ablation study demonstrates that removing either the joint time–frequency scattering transform or the metric learning algorithm noticeably degrades performance.

show abstract

Section: Role Of Dataset Sizementioning

confidence: 76%

Time–frequency scattering accurately models auditory similarities between instrumental playing techniques

Lostanlen

El-Hajj

Rossignol³

et al. 2021

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

show abstract

“…Each element of the fixed length E vector is represented as a product of the posterior and averaged residual vector of each cluster. The collection of the learned cluster centers is a "dictionary," and each learned cluster center is expected to represent characteristics of different regions in the sequence such as onset and overshoot regions, which is expected to help musical instrument recognition [43].…”

Section: B Encoding and Temporal Aggregationmentioning

confidence: 99%

Use of Speaker Recognition Approaches for Learning and Evaluating Embedding Representations of Musical Instrument Sounds

Shi

Cooper

Yamagishi

2022

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Constructing an embedding space for musical instrument sounds that can meaningfully represent new and unseen instruments is important for downstream music generation tasks such as multi-instrument synthesis and timbre transfer. The framework of Automatic Speaker Verification (ASV) provides us with architectures and evaluation methodologies for verifying the identities of unseen speakers, and these can be repurposed for the task of learning and evaluating a musical instrument sound embedding space that can support unseen instruments. Borrowing from state-of-the-art ASV techniques, we construct a musical instrument recognition model that uses a SincNet frontend, a ResNet architecture, and an angular softmax objective function. Experiments on the NSynth and RWC datasets show our model's effectiveness in terms of equal error rate (EER) for unseen instruments, and ablation studies show the importance of data augmentation and the angular softmax objective. Experiments also show the benefit of using a CQT-based filterbank for initializing SincNet over a Mel filterbank initialization. Further complementary analysis of the learned embedding space is conducted with t-SNE visualizations and probing classification tasks, which show that including instrument family labels as a multi-task learning target can help to regularize the embedding space and incorporate useful structure, and that meaningful information such as playing style, which was not included during training, is contained in the embeddings of unseen instruments.

show abstract

“…In the case of tempogram, Spect + Tempo showed improved performance over Han's model. The advantage of onsets in extracting informative cues about musical instrument recognition is proposed in [50]. Human listeners can easily identify instrument sounds from onset portions compared to other portions of the sound.…”

Section: Effect Of Voting and Ablation Study Of Ensemblementioning

confidence: 99%

Transformer-based ensemble method for multiple predominant instruments recognition in polyphonic music

Reghunath

Rajan

2022

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Multiple predominant instrument recognition in polyphonic music is addressed using decision level fusion of three transformer-based architectures on an ensemble of visual representations. The ensemble consists of Mel-spectrogram, modgdgram, and tempogram. Predominant instrument recognition refers to the problem where the prominent instrument is identified from a mixture of instruments being played together. We experimented with two transformer architectures like Vision transformer (Vi-T) and Shifted window transformer (Swin-T) for the proposed task. The performance of the proposed system is compared with that of the state-of-the-art Han’s model, convolutional neural networks (CNN), and deep neural networks (DNN). Transformer networks learn the distinctive local characteristics from the visual representations and classify the instrument to the group where it belongs. The proposed system is systematically evaluated using the IRMAS dataset with eleven classes. A wave generative adversarial network (WaveGAN) architecture is also employed to generate audio files for data augmentation. We train our networks from fixed-length music excerpts with a single-labeled predominant instrument and estimate an arbitrary number of predominant instruments from the variable-length test audio file without any sliding window analysis and aggregation strategy as in existing algorithms. The ensemble voting scheme using Swin-T reports a micro and macro F1 score of 0.66 and 0.62, respectively. These metrics are 3.12% and 12.72% relatively higher than those obtained by the state-of-the-art Han’s model. The architectural choice of transformers with ensemble voting on Mel-spectro-/modgd-/tempogram has merit in recognizing the predominant instruments in polyphonic music.

show abstract

Modeling the onset advantage in musical instrument recognition

Cited by 6 publications

References 25 publications

Time–frequency scattering accurately models auditory similarities between instrumental playing techniques

Time–frequency scattering accurately models auditory similarities between instrumental playing techniques

Use of Speaker Recognition Approaches for Learning and Evaluating Embedding Representations of Musical Instrument Sounds

Transformer-based ensemble method for multiple predominant instruments recognition in polyphonic music

Contact Info

Product

Resources

About