Ethics and Good Practice in Computational Paralinguistics

Batliner, Anton; Hantke, Simone; Schuller, Björn

doi:10.1109/taffc.2020.3021015

Cited by 39 publications

(15 citation statements)

References 133 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although this field has seen tremendous progress in the last decades [1], three major challenges remain for real-world paralinguistics-based SER applications: a) improving on its inferior valence performance [4,8], b) overcoming issues of generalisation and robustness [12,13], and c) alleviating individual-and group-level fairness concerns, which is a prerequisite for ethical emotion recognition technology [14,15]. Previous works have attempted to tackle these issues in isolation, e. g. by using cross-modal knowledge distillation to increase valence performance [16], speech enhancement or data augmentation to improve robustness [12,13], and de-biasing techniques to mitigate unfair outcomes [17].…”

Section: Introductionmentioning

confidence: 99%

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Wagner¹,

Triantafyllopoulos²,

Wierstorf³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent advances in transformer-based architectures which are pre-trained in self-supervised manner have shown great promise in several machine learning tasks. In the audio domain, such architectures have also been successfully utilised in the field of speech emotion recognition (SER). However, existing works have not evaluated the influence of model size and pre-training data on downstream performance, and have shown limited attention to generalisation, robustness, fairness, and efficiency. The present contribution conducts a thorough analysis of these aspects on several pre-trained variants of wav2vec 2.0 and HuBERT that we fine-tuned on the dimensions arousal, dominance, and valence of MSP-Podcast, while additionally using IEMOCAP and MOSI to test cross-corpus generalisation. To the best of our knowledge, we obtain the top performance for valence prediction without use of explicit linguistic information, with a concordance correlation coefficient (CCC) of .638 on MSP-Podcast. Furthermore, our investigations reveal that transformer-based architectures are more robust to small perturbations compared to a CNN-based baseline and fair with respect to biological sex groups, but not towards individual speakers. Finally, we are the first to show that their extraordinary success on valence is based on implicit linguistic information learnt during finetuning of the transformer layers, which explains why they perform on-par with recent multimodal approaches that explicitly utilise textual information. Our findings collectively paint the following picture: transformer-based architectures constitute the new state-of-the-art in SER, but further advances are needed to mitigate remaining robustness and individual speaker issues. To make our findings reproducible, we release the best performing model to the community.

show abstract

Section: Introductionmentioning

confidence: 99%

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Wagner¹,

Triantafyllopoulos²,

Wierstorf³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Consequently, significantly outperforming the MSC baseline system (71.8 %, 11 012 Test set samples) at a significance level of

requires at least an absolute improvement of

. Note that Null-Hypothesis-Testing with

-values as criterion has been criticised from its beginning; see the statement of the American Statistical Association in [68] , [69] . Therefore, we provide this plot with

-values as a service for readers interested in this approach, not as a guideline for deciding between approaches.…”

Section: Challenge Results and Contributionsmentioning

confidence: 99%

Face mask recognition from audio: The MASC database and an overview on the mask challenge

Mohamed

Nessiem

Batliner

et al. 2022

Pattern Recognition

Self Cite

View full text Add to dashboard Cite

The sudden outbreak of COVID-19 has resulted in tough challenges for the field of biometrics due to its spread via physical contact, and the regulations of wearing face masks. Given these constraints, voice biometrics can offer a suitable contact-less biometric solution; they can benefit from models that classify whether a speaker is wearing a mask or not. This article reviews the Mask Sub-Challenge (MSC) of the INTERSPEECH 2020 COMputational PARalinguistics challengE (ComParE), which focused on the following classification task: Given an audio chunk of a speaker, classify whether the speaker is wearing a mask or not. First, we report the collection of the Mask Augsburg Speech Corpus (MASC) and the baseline approaches used to solve the problem, achieving a performance of Unweighted Average Recall (UAR). We then summarise the methodologies explored in the submitted and accepted papers that mainly used two common patterns: (i) phonetic-based audio features, or (ii) spectrogram representations of audio combined with Convolutional Neural Networks (CNNs) typically used in image processing. Most approaches enhance their models by adapting ensembles of different models and attempting to increase the size of the training data using various techniques. We review and discuss the results of the participants of this sub-challenge, where the winner scored a UAR of . Moreover, we present the results of fusing the approaches, leading to a UAR of . Finally, we present a smartphone app that can be used as a proof of concept demonstration to detect in real-time whether users are wearing a face mask; we also benchmark the run-time of the best models.

show abstract

“…Further concerns in this context will discuss legal and societal implications. All of these cannot be discussed here-rather, we can provide pointers for the interested reader as starting points (50)(51)(52)(53)(54)(55).…”

Section: Ethicsmentioning

confidence: 99%

“…Next, one must assure that common points of reference for comparison across studies are given, the aim of an audio task is well-decided upon, results are interpretable, and communicated to all, including in particular communication of potential limitations ( 50 ). Further concerns in this context will discuss legal and societal implications.…”

Section: Challengesmentioning

confidence: 99%

COVID-19 and Computer Audition: An Overview on What Speech & Sound Analysis Could Contribute in the SARS-CoV-2 Corona Crisis

Schuller

Qian

et al. 2021

Front. Digit. Health

View full text Add to dashboard Cite

At the time of writing this article, the world population is suffering from more than 2 million registered COVID-19 disease epidemic-induced deaths since the outbreak of the corona virus, which is now officially known as SARS-CoV-2. However, tremendous efforts have been made worldwide to counter-steer and control the epidemic by now labelled as pandemic. In this contribution, we provide an overview on the potential for computer audition (CA), i.e., the usage of speech and sound analysis by artificial intelligence to help in this scenario. We first survey which types of related or contextually significant phenomena can be automatically assessed from speech or sound. These include the automatic recognition and monitoring of COVID-19 directly or its symptoms such as breathing, dry, and wet coughing or sneezing sounds, speech under cold, eating behaviour, sleepiness, or pain to name but a few. Then, we consider potential use-cases for exploitation. These include risk assessment and diagnosis based on symptom histograms and their development over time, as well as monitoring of spread, social distancing and its effects, treatment and recovery, and patient well-being. We quickly guide further through challenges that need to be faced for real-life usage and limitations also in comparison with non-audio solutions. We come to the conclusion that CA appears ready for implementation of (pre-)diagnosis and monitoring tools, and more generally provides rich and significant, yet so far untapped potential in the fight against COVID-19 spread.

show abstract

Ethics and Good Practice in Computational Paralinguistics

Cited by 39 publications

References 133 publications

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Face mask recognition from audio: The MASC database and an overview on the mask challenge

COVID-19 and Computer Audition: An Overview on What Speech & Sound Analysis Could Contribute in the SARS-CoV-2 Corona Crisis

Contact Info

Product

Resources

About