Amplitude-Frequency Analysis of Emotional Speech Using Transfer Learning and Classification of Spectrogram Images

Abstract

Automatic speech emotion recognition (SER) techniques based on acoustic analysis show high confusion between certain emotional categories. This study used an indirect approach to provide insights into the amplitude-frequency characteristics of different emotions in order to support the development of future, more efficiently differentiating SER methods. The analysis was carried out by transforming short 1-second blocks of speech into RGB or grey-scale images of spectrograms. The images were used to fine-tune a pre-trained image classification network to recognize emotions. Spectrogram representation on four different frequency scales-linear, melodic, equivalent rectangular bandwidth (ERB), and logarithmic-allowed observation of the effects of high, mid-high, mid-low and low frequency characteristics of speech, respectively. Whereas the use of either red (R), green (G) or blue (B) components of RGB images showed the importance of speech components with high, mid and low amplitude levels, respectively. Experiments conducted on the Berlin emotional speech (EMO-DB) data revealed the relative positions of seven emotional categories (anger, boredom, disgust, fear, joy, neutral and sadness) on the amplitudefrequency plane.

Emotional sounds of crowds: spectrogram-based analysis using deep learning

“…Since, to the best of our knowledge, this work represents the first attempt to use a spectrogram-based approach with crowd emotional sounds, there are no datasets or results in the literature to compare our results. To give a general idea on the algorithm performance, we can provide a coarse-grained comparison with the same approach applied to individual-speech emotional sound [ 25 ], aware that the specific features of the two cases are not strictly comparable. On crowd sounds, the performances are improved on average of 10% .…”

“…Such choice can potentially affect the analysis results because different scales emphasize different frequency ranges, thus different components of the sounds in a crowd sound block tend to emerge with different scales. In this work, we systematically analyze four frequency scales, which have been chosen for their intrinsic characteristics, expressing the different contribution in increasingly lower frequency ranges: Mel [ 34 ], for the 4–6 kHz range Erb [ 28 ], for the 2–4 kHz range Bark [ 37 ], for the 0–3.5 kHz range Log [ 25 ], for the 0.02-2 kHz range …”

“…The first approach follows the standard practices in image classification, as used in state-of-the-art works on speech emotion recognition [ 25 ], where a dataset is partitioned in two subsets by randomly picking images and assigning them to the training set and test set , according to a given proportion. In the used training/test set ratio, 80% of images are assigned to the training set and 20% to the test set.…”

“…A general comparison has been discussed, by comparing the experimental results for crowd sound emotion with the result available for a similar algorithm on a different emotional model, i.e. on the domain of individual-speech emotions studied in [ 25 ]. Note that our domain and emotional model show a 10% improvement with respect to the 80% average accuracy of the individual-speech domain on every scale, even in the second experimental setting where we lower the performance gaining a better consistency.…”

“…The new levels weights are adjusted by a supervised domain-specific fine-tuning phase [ 2 – 4 , 19 , 31 ]. It is worth noticing that TL methodology is not new to sound recognition [ 30 , 35 ], but its application to emotional speech is due to recent works [ 25 ]. To the best of our knowledge, this work and the preliminary [ 17 ] represent the first application to crowd sound emotion recognition.…”