An Evolutionary-based Generative Approach for Audio Data Augmentation

Mertes, Silvan; Baird, Alice; Schiller, Dominik; Schuller, Björn W.; André, Elisabeth

doi:10.1109/mmsp48831.2020.9287156

Cited by 21 publications

(15 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the DEEPSPECTRUM features, we extract a 2,560 dimensional feature set of deep data-representations using the DEEPSPECTRUM toolkit (Amiriparian et al, 2017). DEEPSPECTRUM has shown success for various audio-and speech-based tasks (Mertes et al, 2020), and extracts features from the audio data using pre-trained convolutional neural networks. For this study, we extract features based on the viridis colour map, and the deep features are extracted from the layer fc7 of AlexNet (Krizhevsky et al, 2012).…”

Section: Featuresmentioning

confidence: 99%

An Evaluation of Speech-Based Recognition of Emotional and Physiological Markers of Stress

Baird

Triantafyllopoulos

Zänkert

et al. 2021

Front. Comput. Sci.

Self Cite

View full text Add to dashboard Cite

Life in modern societies is fast-paced and full of stress-inducing demands. The development of stress monitoring methods is a growing area of research due to the personal and economic advantages that timely detection provides. Studies have shown that speech-based features can be utilised to robustly predict several physiological markers of stress, including emotional state, continuous heart rate, and the stress hormone, cortisol. In this contribution, we extend previous works by the authors, utilising three German language corpora including more than 100 subjects undergoing a Trier Social Stress Test protocol. We present cross-corpus and transfer learning results which explore the efficacy of the speech signal to predict three physiological markers of stress—sequentially measured saliva-based cortisol, continuous heart rate as beats per minute (BPM), and continuous respiration. For this, we extract several features from audio as well as video and apply various machine learning architectures, including a temporal context-based Long Short-Term Memory Recurrent Neural Network (LSTM-RNN). For the task of predicting cortisol levels from speech, deep learning improves on results obtained by conventional support vector regression—yielding a Spearman correlation coefficient (ρ) of 0.770 and 0.698 for cortisol measurements taken 10 and 20 min after the stress period for the two corpora applicable—showing that audio features alone are sufficient for predicting cortisol, with audiovisual fusion to an extent improving such results. We also obtain a Root Mean Square Error (RMSE) of 38 and 22 BPM for continuous heart rate prediction on the two corpora where this information is available, and a normalised RMSE (NRMSE) of 0.120 for respiration prediction (−10: 10). Both of these continuous physiological signals show to be highly effective markers of stress (based on cortisol grouping analysis), both when available as ground truth and when predicted using speech. This contribution opens up new avenues for future exploration of these signals as proxies for stress in naturalistic settings.

show abstract

Section: Featuresmentioning

confidence: 99%

An Evaluation of Speech-Based Recognition of Emotional and Physiological Markers of Stress

Baird

Triantafyllopoulos

Zänkert

et al. 2021

Front. Comput. Sci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…LVE was applied to different domains, such as video games [30] or fingerprint-based biometric systems [4]. It was also deployed successfully for searching through the latent space of a WaveGAN for the purpose of augmenting datasets [21]. Further, LVE was used to give human users the ability to interactively evolve through a learned GAN space [3,31].…”

Section: Gans and Controllabilitymentioning

confidence: 99%

Taming the Chaos: Exploring Graphical Input Vector Manipulation User Interfaces for GANs in a Musical Context

2021

Self Cite

View full text Add to dashboard Cite

Generative Adversarial Networks (GANs) are a widely used tool for generating highly realistic artificial data. As the output of these networks can show high diversity and novelty, GANs have the potential to be used as creative tools. However, using GANs in this context poses major challenges due to their unpredictability and lack of controllability, making it difficult for creative people to realize their artistic vision. To address this problem, we present two graphical user interfaces that visually order the (otherwise chaotic) latent input space of a GAN that was trained to generate drum samples. Further, these GUIs provide convergent search functions that allow users to fine-tune generated sounds. By doing so, we provide the ability to create sounds more purposefully to soundaffine users such as musicians or sound engineers. Additionally, we present the results of a user study that we conducted in order to explore our approach in accuracy-oriented and creative tasks. Our results indicate that usability and pragmatic qualities play a more important role for users than aesthetic-oriented aspects.Although not improving the accuracy within reproductive tasks, we observed that convergent search functions, if available, were used significantly more often than divergent/randomized search functions. CCS CONCEPTS• Human-centered computing → User interface design; Empirical studies in interaction design.

show abstract

“…We utilise WAVEGAN to generate new audio data, as first proposed in [4]. We have chosen WAVEGAN , as it shows promise for a range of audio generation tasks, in the domain of emotional speech [21], and music [4], as well as being successfully adapted for the task of data augmentation [22]. Typically, in a GAN paradigm, a generator produces new samples and competes against the discriminator, attempting to classify the instances as fake or real.…”

Section: Audio Generationmentioning

confidence: 99%

A Prototypical Network Approach for Evaluating Generated Emotional Speech

Baird¹,

Mertes²,

Milling³

et al. 2021

Interspeech 2021

Self Cite

View full text Add to dashboard Cite

The collection of emotional speech data is a time-consuming and costly endeavour. Generative networks can be applied to augment the limited audio data artificially. However, it is challenging to evaluate generated audio for its similarity to source data, as current quantitative metrics are not necessarily suited to the audio domain. We explore the use of a prototypical network to evaluate four classes of generated emotional audio with this in mind. We first extract spectrogram images from WAVEGAN generated audio and other audio augmentation approaches, comparing similarity to the class prototype and diversity within the embedding space. Furthermore, we augment the source training set with each augmentation type and perform a classification to explore the generated audio plausibility. Results suggest that quality and diversity can be quantitatively observed with this approach. In the chosen context, we see that WAVEGAN generated data is recognisable as a source data class (F1-score 43.6 %), and the samples add similar diversity as unseen source data. This result leads to more plausible data for augmentation of the source training set -achieving up to 63.9 % F1 which is a 3.5 % improvement over the source data baseline.

show abstract

An Evolutionary-based Generative Approach for Audio Data Augmentation

Cited by 21 publications

References 19 publications

An Evaluation of Speech-Based Recognition of Emotional and Physiological Markers of Stress

An Evaluation of Speech-Based Recognition of Emotional and Physiological Markers of Stress

Taming the Chaos: Exploring Graphical Input Vector Manipulation User Interfaces for GANs in a Musical Context

A Prototypical Network Approach for Evaluating Generated Emotional Speech

Contact Info

Product

Resources

About