Multimodal music emotion recognition in Indonesian songs based on CNN-LSTM, XLNet transformers

Abstract

Music carries emotional information and allows the listener to feel the emotions contained in the music. This study proposes a multimodal music emotion recognition (MER) system using Indonesian song and lyrics data. In the proposed multimodal system, the audio data will use the mel spectrogram feature, and the lyrics feature will be extracted by going through the tokenizing process from XLNet. Convolutional long short term memory network (CNN-LSTM) performs the audio classification task, while XLNet transformers performs the lyrics classification task. The outputs of the two classification tasks are probability weight and actual prediction with the value of positive, neutral, and negative emotions, which are then combined using the stacking ensemble method. The combined output will be trained into an artificial neural network (ANN) model to get the best probability weight output. The multimodal system achieves the best performance with an accuracy of 80.56%. The results showed that the multimodal method of recognizing musical emotions gave better performance than the single modal method. In addition, hyperparameter tuning can affect the performance of multimodal systems.

The Analysis of Music Emotion and Visualization Fusing Long Short-Term Memory Networks Under the Internet of Things

“…Sams et al used convolutional LSTM networks to perform audio classification tasks. The results suggested that the multimodal method for music emotion recognition performed better than the single-modal method (Sams & Zahra, 2023). This is consistent with the research on music emotion and visualization of the fusion of LSTM networks supported by the IoT in this paper.…”

MERP: A Music Dataset with Emotion Ratings and Raters’ Profile Information

“…The models are simple in design and intended to be supplementary performance benchmarks on the dataset. In future work, more state-of-the-art methods such as convolutional neural networks [ 70 , 71 , 72 , 73 , 74 , 75 , 76 ], or transformer architectures [ 60 , 77 , 78 ] could be used with the dataset for further experimentation with profile information and its uses for building improved MER models.…”

Deep learning and acoustic approach for mechanical failure detection in industrial machinery

“…Long short-term memory (LSTM), gate repeating unit (GRU), autoencoder (AE), and convolutional neural network (CNN) are some examples of deep learning algorithms for classification. Long short-term memory (LSTM) is one of the deep learning algorithms that can be used for classification, prediction, and control [8][9][10][11]. It can learn complex patterns and relationships in the input data, making it a valuable tool for a wide range of tasks in various fields such as finance, healthcare, and natural language processing [12][13][14].…”