Transformer-based ensemble method for multiple predominant instruments recognition in polyphonic music

Abstract

Multiple predominant instrument recognition in polyphonic music is addressed using decision level fusion of three transformer-based architectures on an ensemble of visual representations. The ensemble consists of Mel-spectrogram, modgdgram, and tempogram. Predominant instrument recognition refers to the problem where the prominent instrument is identified from a mixture of instruments being played together. We experimented with two transformer architectures like Vision transformer (Vi-T) and Shifted window transformer (Swin-T) for the proposed task. The performance of the proposed system is compared with that of the state-of-the-art Han’s model, convolutional neural networks (CNN), and deep neural networks (DNN). Transformer networks learn the distinctive local characteristics from the visual representations and classify the instrument to the group where it belongs. The proposed system is systematically evaluated using the IRMAS dataset with eleven classes. A wave generative adversarial network (WaveGAN) architecture is also employed to generate audio files for data augmentation. We train our networks from fixed-length music excerpts with a single-labeled predominant instrument and estimate an arbitrary number of predominant instruments from the variable-length test audio file without any sliding window analysis and aggregation strategy as in existing algorithms. The ensemble voting scheme using Swin-T reports a micro and macro F1 score of 0.66 and 0.62, respectively. These metrics are 3.12% and 12.72% relatively higher than those obtained by the state-of-the-art Han’s model. The architectural choice of transformers with ensemble voting on Mel-spectro-/modgd-/tempogram has merit in recognizing the predominant instruments in polyphonic music.

Augmentation Embedded Deep Convolutional Neural Network for Predominant Instrument Recognition

“…In their work, they propose a method to recognize both pitches and instruments [16]. To augment the data, they employed a Wave Generative Adversarial Network (WaveGAN) architecture to generate audio files [7][8][9]. These approaches demonstrate the utilization of various techniques, including feature extraction, deep learning, image processing, and data augmentation, to improve instrument recognition accuracy and handle challenges such as low-quality recordings and polyphonic music.…”

“…F1 Micro F1 Macro SVM [25] 0.36 0.27 Bosch et al [26] 0.50 0.43 MTF-DNN (2018) [27] 0.32 0.28 Audio DNN [28] 0.55 0.51 ConvNet (2017) [15] 0.62 0.52 Muti-task ConvNet (2020) [10] 0.66 0.58 Kratimenos et al (2021) [14] 0.65 0.55 WaveGAN ConvNet (2021) [7] 0.65 0.60 Voting-Swin-T (2022) [8] 0 According to the results presented in Table 5, the proposed AEDCN model outperforms other neural network-based models as well as data augmentation-based instrument recognition models.…”

“…In the first stage, data augmentation techniques like Generative Adversarial Networks (GANs) are utilized to generate additional training data. In the second stage, deep neural networks such as Convolutional Neural Networks (CNNs), Gate Recurrent Units (GRUs), Convolutional Gate Recurrent Units (Conv GRUs), and Transformers are employed to map the augmented data to instrument labels [7][8][9]. Additionally, researchers have explored label augmentation, which involves constructing auxiliary classifiers based on additional labels introduced within a multi-task learning framework.…”