Deep-Learning-Based Multimodal Emotion Classification for Music Videos

Pandeya, Yagya Raj; Bhattarai, Bhuwan; Lee, Joonwhoan

doi:10.3390/s21144927

Cited by 44 publications

(17 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Moreover, the traditional machine learning methods also face the problems of high cost of audio feature extraction and inability to deal with large samples. Based on this, researchers began to try using deep learning methods such as Gaussian mixture model (GMM) [8] and convolutional neural network (CNN) [9] to classify music emotion. At the same time, in order to solve the problem of low accuracy of traditional deep learning methods, researchers began to seek some composite methods to classify music emotion more accurately.…”

Section: Introductionmentioning

confidence: 99%

Research on Music Emotional Expression Based on Reinforcement Learning and Multimodal Information

Li-ge

Tian

2022

Mobile Information Systems

View full text Add to dashboard Cite

With the continuous development of the research in the field of emotion analysis, music, as a common multimodal information carrier in people’s daily life, often transmits emotion through lyrics and melody, so it has been gradually incorporated into the research category of emotion analysis. The fusion classification model based on CNN-LSTM proposed in this paper effectively improves the accuracy of emotional classification of audio and lyrics. At the same time, in view of the problem that the traditional decision-level fusion method ignores the correlation between modes and the limitations of dataset, this paper further improves the existing Thayer dimension emotional decision fusion method, takes the audio energy axis data as the main discrimination basis, and improves the accuracy of decision fusion classification. Based on the results of music emotion analysis, this paper further carries out the task of music generation. Based on the feature that there is often consistent emotional expression between music words and songs, a dual Seq2Seq framework based on reinforcement learning is constructed. By introducing the reward value of emotional consistency and content fidelity, the output melody has the same emotion with the input lyrics and good results are achieved. Compared with the ordinary Seq2Seq, the accuracy of our proposed model is improved by about 1.1%. This shows that the accuracy of the model can be effectively improved by using reinforcement learning.

show abstract

Section: Introductionmentioning

confidence: 99%

Research on Music Emotional Expression Based on Reinforcement Learning and Multimodal Information

Li-ge

Tian

2022

Mobile Information Systems

View full text Add to dashboard Cite

show abstract

“…(4) Singer identi cation and classi cation identify and classify the singers of unknown songs. (5) Emotion recognition and classi cation identify and classify the emotion types expressed by songs or pure music. Among them, identifying and classifying music genres are very important in searching for music information.…”

Section: Introductionmentioning

confidence: 99%

“…Among them, identifying and classifying music genres are very important in searching for music information. Many music users are only interested in a particular type of music, and the role of the music type recognition and classi cation system is to classify music according to its style [3][4][5]. So that they can recommend music according to their interests and hobbies, which is convenient for users to quickly retrieve and e ciently manage happy music.…”

Section: Introductionmentioning

confidence: 99%

Feature Extraction and Classification of Music Content Based on Deep Learning

Shi¹,

2022

Advances in Multimedia

View full text Add to dashboard Cite

To study the use of in-depth training in extracting and classifying the content of music samples, the work offers an algorithm for identifying and classifying musical genres based on a deep network of beliefs, enabling it to be used to extract and classify traditional Chinese musical instruments, and using real-world experiments to test its performance after training. The experimental results are as follows: the improved depth confidence network algorithm has the highest accuracy for music recognition and classification, which can reach 75.8%, higher than other traditional methods. The improved depth confidence network identifies and classifies Chinese traditional musical instruments through Softmax layer, and the accuracy is even as high as 99.2%; DBN is combined with Softmax neural network algorithm when only a few labeled samples in the training set are used for network fine-tuning, and the accuracy of the algorithm can still reach more than 90%, which can reduce the workload in the early stage. This study effectively solves the problem of too much workload and low accuracy in the process of music content recognition, classification, and extraction.

show abstract

“…Using conjoint model based on the fusion of a different group of data’s sensors has been introduced in emotion recognition by Pandeya, Y. [ 23 ]. It uses the deep learning to classify the emotions [ 23 ] from audio and video information.…”

Section: Introductionmentioning

confidence: 99%

“…[ 23 ]. It uses the deep learning to classify the emotions [ 23 ] from audio and video information. Rueping, S. proposed subgroup ranking using the support vector machine (SVM) to rank subgroups with respect to the user’s concept of interestingness [ 24 ].…”

Section: Introductionmentioning

confidence: 99%

Subgroup Preference Neural Network

Elgharabawy

Prasad

Lin

2021

Sensors

View full text Add to dashboard Cite

Subgroup label ranking aims to rank groups of labels using a single ranking model, is a new problem faced in preference learning. This paper introduces the Subgroup Preference Neural Network (SGPNN) that combines multiple networks have different activation function, learning rate, and output layer into one artificial neural network (ANN) to discover the hidden relation between the subgroups’ multi-labels. The SGPNN is a feedforward (FF), partially connected network that has a single middle layer and uses stairstep (SS) multi-valued activation function to enhance the prediction’s probability and accelerate the ranking convergence. The novel structure of the proposed SGPNN consists of a multi-activation function neuron (MAFN) in the middle layer to rank each subgroup independently. The SGPNN uses gradient ascent to maximize the Spearman ranking correlation between the groups of labels. Each label is represented by an output neuron that has a single SS function. The proposed SGPNN using conjoint dataset outperforms the other label ranking methods which uses each dataset individually. The proposed SGPNN achieves an average accuracy of 91.4% using the conjoint dataset compared to supervised clustering, decision tree, multilayer perceptron label ranking and label ranking forests that achieve an average accuracy of 60%, 84.8%, 69.2% and 73%, respectively, using the individual dataset.

show abstract

Deep-Learning-Based Multimodal Emotion Classification for Music Videos

Cited by 44 publications

References 72 publications

Research on Music Emotional Expression Based on Reinforcement Learning and Multimodal Information

Research on Music Emotional Expression Based on Reinforcement Learning and Multimodal Information

Feature Extraction and Classification of Music Content Based on Deep Learning

Subgroup Preference Neural Network

Contact Info

Product

Resources

About