Multimodal Music Emotion Recognition Method Based on the Combination of Knowledge Distillation and Transfer Learning

Abstract

The main difficulty of music emotion recognition is the lack of sufficient labeled data. Only the labeled data with unbalanced categories are used to train the emotion recognition model. Not only is accurate labeling of emotion categories costly and time-consuming, but it also requires extensive musical background for labelers At the same time, the emotion of music is often affected by many factors. Singing methods, music styles, arrangement methods, lyrics, and other factors will affect the expression of music emotions. This paper proposes a multimodal method based on the combination of knowledge distillation and music style transfer learning and verifies the effectiveness of the method on 20,000 songs. Experiments show that compared with traditional methods, such as single audio, single lyric, and single audio with multimodal lyric methods, the method proposed in this paper has significantly improved the accuracy of emotion recognition, and the generalization ability has been significantly improved.

Multimodal music emotion recognition in Indonesian songs based on CNN-LSTM, XLNet transformers

“…Then, we need to transform the audio signal into the Mel-spectrogram using the Librosa library. Several studies also use Mel-spectrogram and achieved great results [4], [10], [22], [23]. After all the song data was turned into a spectrogram, all the data was grouped into positive, neutral, and negative and saved to a folder then uploaded to Google drive.…”