Dimensional Affect Uncertainty Modelling for Apparent Personality Recognition

Tellamekala, Mani Kumar; Giesbrecht, Timo; Valstar, Michel

doi:10.1109/taffc.2022.3189974

“…The label uncertainty cannot be reduced even if more data-samples are collected. Whereas, model uncertainty, or epistemic uncertainty, accounts for the uncertainty in model parameters, and the resulting uncertainty can be explained given enough data-samples [15]- [17]. Label uncertainty has been further categorized into arXiv:2209.15449v1 [eess.AS] 30 Sep 2022 homoscedastic uncertainty, which remains constant across data-samples, and heteroscedastic uncertainty, whose uncertainty depends on the respective data-sample.…”

Section: Introductionmentioning

confidence: 99%

End-to-End Label Uncertainty Modeling in Speech Emotion Recognition using Bayesian Neural Networks and Label Distribution Learning

Prabhu¹,

Lehmann‐Willenbrock²,

Gerkmann³

2022

Preprint

1

0

View full text Add to dashboard Cite

To train machine learning algorithms to predict emotional expressions in terms of arousal and valence, annotated datasets are needed. However, as different people perceive others’ emotional expressions differently, their annotations are per se subjective. For this, annotations are typically collected from multiple annotators and averaged to obtain ground-truth labels. However, when exclusively trained on this averaged ground-truth, the trained network is agnostic to the inherent subjectivity in emotional expressions. In this work, we therefore propose an end-to-end Bayesian neural network capable of being trained on a distribution of labels to also capture the subjectivity-based label uncertainty. Instead of a Gaussian, we model the label distribution using Student’s t-distribution, which also accounts for the number of annotations. We derive the corresponding Kullback-Leibler divergence loss and use it to train an estimator for the distribution of labels, from which the mean and uncertainty can be inferred. We validate the proposed method using two in-the-wild datasets. We show that the proposed t-distribution based approach achieves state-of-the-art uncertainty modeling results in speech emotion recognition, and also consistent results in cross-corpora evaluations. Furthermore, analyses reveal that the advantage of a t-distribution over a Gaussian grows with increasing inter-annotator correlation and a decreasing number of annotators.

show abstract

“…Proposed BBB-LDL versions: We use three versions of the proposed label uncertainty model. Firstly, the Model Uncertainty (MU) version, which share the same DNN architecture as the other BBB version but trained on (17) with α = 0. Secondly, the Label Uncertainty (MU+LU) version with also captures the label uncertainty and trained on (17) with α = 1.…”

Section: E2e Baselinementioning

confidence: 99%

“…Firstly, the Model Uncertainty (MU) version, which share the same DNN architecture as the other BBB version but trained on (17) with α = 0. Secondly, the Label Uncertainty (MU+LU) version with also captures the label uncertainty and trained on (17) with α = 1. The MU+LU version however makes a Gaussian assumption on Y t , thereby L KL follows (12).…”

Section: E2e Baselinementioning

confidence: 99%

“…The MU+LU version however makes a Gaussian assumption on Y t , thereby L KL follows (12). Finally, the t-distribution Label Uncertainty (t-LU) version, which is trained on the same loss function (17) but models Y t as a t-distribution, and L KL follows (16).…”

Section: E2e Baselinementioning

confidence: 99%

“…The label uncertainty cannot be reduced even if more data-samples are collected. Whereas, model uncertainty, or epistemic uncertainty, accounts for the uncertainty in model parameters, and the resulting uncertainty can be explained given enough data-samples [15]- [17]. Label uncertainty has been further categorized into homoscedastic uncertainty, which remains constant across data-samples, and heteroscedastic uncertainty, whose uncertainty depends on the respective data-sample.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

End-to-End Label Uncertainty Modeling in Speech Emotion Recognition using Bayesian Neural Networks and Label Distribution Learning

Prabhu¹,

Lehmann‐Willenbrock²,

Gerkmann³

2022

Preprint

0

View full text Add to dashboard Cite

To train machine learning algorithms to predict emotional expressions in terms of arousal and valence, annotated datasets are needed. However, as different people perceive others’ emotional expressions differently, their annotations are per se subjective. For this, annotations are typically collected from multiple annotators and averaged to obtain ground-truth labels. However, when exclusively trained on this averaged ground-truth, the trained network is agnostic to the inherent subjectivity in emotional expressions. In this work, we therefore propose an end-to-end Bayesian neural network capable of being trained on a distribution of labels to also capture the subjectivity-based label uncertainty. Instead of a Gaussian, we model the label distribution using Student’s t-distribution, which also accounts for the number of annotations. We derive the corresponding Kullback-Leibler divergence loss and use it to train an estimator for the distribution of labels, from which the mean and uncertainty can be inferred. We validate the proposed method using two in-the-wild datasets. We show that the proposed t-distribution based approach achieves state-of-the-art uncertainty modeling results in speech emotion recognition, and also consistent results in cross-corpora evaluations. Furthermore, analyses reveal that the advantage of a t-distribution over a Gaussian grows with increasing inter-annotator correlation and a decreasing number of annotators.

show abstract

Challenges and Emerging Trends for Machine Reading of the Mind from Facial Expressions

Ghazouani

2023

SN COMPUT. SCI.

1

0

View full text Add to dashboard Cite

Dimensional Affect Uncertainty Modelling for Apparent Personality Recognition

Cited by 5 publications

References 43 publications

End-to-End Label Uncertainty Modeling in Speech Emotion Recognition using Bayesian Neural Networks and Label Distribution Learning

End-to-End Label Uncertainty Modeling in Speech Emotion Recognition using Bayesian Neural Networks and Label Distribution Learning

End-to-End Label Uncertainty Modeling in Speech Emotion Recognition using Bayesian Neural Networks and Label Distribution Learning

Challenges and Emerging Trends for Machine Reading of the Mind from Facial Expressions

Contact Info

Product

Resources

About