Emotional Video to Audio Transformation Using Deep Recurrent Neural Networks and a Neuro-Fuzzy System

Abstract

Generating music with emotion similar to that of an input video is a very relevant issue nowadays. Video content creators and automatic movie directors bene t from maintaining their viewers engaged, which can be facilitated by producing novel material eliciting stronger emotions in them. Moreover, there is currently a demand for more empathetic computers to aid humans in applications such as augmenting the perception ability of visually-and/or hearing-impaired people. Current approaches overlook the video's emotional characteristics in the music generation step, only consider static images instead of videos, are unable to generate novel music, and require a high level of human e ort and skills. In this study, we propose a novel hybrid deep neural network that uses an Adaptive Neuro-Fuzzy Inference System to predict a video's emotion from its visual features and a deep Long Short-Term Memory Recurrent Neural Network to generate its corresponding audio signals with similar emotional inkling. e former is able to appropriately model emotions due to its fuzzy properties, and the latter is able to model data with dynamic time properties well due to the availability of the previous hidden state information. e novelty of our proposed method lies in the extraction of visual emotional features in order to transform them into audio signals with corresponding emotional aspects for users. Quantitative experiments show low mean absolute errors of 0.217 and 0.255 in the Lindsey and DEAP datasets, respectively, and similar global features in the spectrograms.is indicates that our model is able to appropriately perform domain transformation between visual and audio features. Based on experimental results, our model can e ectively generate an audio that matches the scene eliciting a similar emotion from the viewer in both datasets, and music generated by our model is also chosen more often (code available online at https://github.com/gcunhase/Emotional-Video-to-Audio-with-ANFIS-DeepRNN).Hinda i Ma hema ical P oblem in Enginee ing Vol me 2020, A icle ID 8478527, 15 age h ://doi.o g/

Analyzing the Role of Emotional Intelligence on the Performance of Small and Medium Enterprises (SMEs) Using AI-Based Convolutional Neural Networks (CNNs)

“…e proposed hybrid DNN introduces an adaptive neuro-fuzzy inference method to anticipate a video's mood. Audio waves with a similar emotional experience may be generated using visual signals and a deep short-term memory recurring neural network [8]. Its fuzzy features allow to accurately depicting emotions, while its ability to model data with dynamic time qualities makes it a good fit for data with dynamic temporal properties.…”

Deep Neuro-Fuzzy System application trends, challenges, and future perspectives: a systematic survey

“…(vii) DNFS application on video classification and robotics The study of Nguyen et al ( 2019 ) presented a novel convolutional neuro-fuzzy network, which incorporated a CNN into the fuzzy logic domain to derive high-level features of emotions from the text, audio, and image data. Alternatively, the authors in (Cunha Sergio and Lee 2020 ) proposed a novel hybrid DNN with ANFIS to interpret the emotions of a video from its visual features and a deep long short-term memory recurrent neural network to produce the related audio signals with an equal emotional impression. Likewise, in another study (Savchenko et al 2018 ), the authors implemented a fuzzy analysis with a CNN in still-to-video recognition.…”