Overlapped speech and gender detection with WavLM pre-trained features

Abstract

This article focuses on overlapped speech and gender detection in order to study interactions between women and men in French audiovisual media (Gender Equality Monitoring project). In this application context, we need to automatically segment the speech signal according to speakers gender, and to identify when at least two speakers speak at the same time. We propose to use WavLM model which has the advantage of being pre-trained on a huge amount of speech data, to build an overlapped speech detection (OSD) and a gender detection (GD) systems. In this study, we use two different corpora. The DIHARD III corpus which is well adapted for the OSD task but lack gender information. The ALLIES corpus fits with the project application context. Our best OSD system is a Temporal Convolutional Network (TCN) with WavLM pre-trained features as input, which reaches a new state-of-the-art F1-score performance on DIHARD. A neural GD is trained with WavLM inputs on a gender balanced subset of the French broadcast news ALLIES data, and obtains an accuracy of 94.9%. This work opens new perspectives for human science researchers regarding the differences of representation between women and men in French media.

An Explainable Proxy Model for Multilabel Audio Segmentation

“…The proxy models are expected to reach similar, eventually slightly lower, performances as the teacher. The score obtained on OSD by the teacher model compares with the state of the art which is 63.4% on DiHard III [14].…”

“…The teacher model is similar to [14] and is composed of two main parts: feature extraction and sequence modeling. The former is performed using pre-trained Wavlm Large [22] that outputs a sequence of 1024-dimension vectors.…”

“…Currently, the segmentation is mainly performed with neural networks and supervised learning. While each task has been generally solved independently as a binary frame-wise classification task (SAD [12], OSD [13,14], MD [15,16]), more recent approaches propose to solve multiple tasks simultaneously. The multiclass model predicts a single class and class intersection is empty.…”

“…For example, in [17], authors propose to segment speech, music, and noise with a single multiclass model. A few works also report joint SAD and OSD [18][19][20]. In this paper, SAD, OSD, MD, and ND are simultaneously solved as a multilabel frame classification task.…”

Microphone Array Channel Combination Algorithms for Overlapped Speech Detection

“…Since overlapping speech is a rare event, the classes are unbalanced [57]. The class balance can be improved by artificially generating additional overlapped data by combining single-speaker utterances from other datasets [18,39] or random segments of the training data at training time [8].…”