Triplet Network with Attention for Speaker Diarization

Abstract

In automatic speech processing systems, speaker diarization is a crucial front-end component to separate segments from different speakers. Inspired by the recent success of deep neural networks (DNNs) in semantic inferencing, triplet loss-based architectures have been successfully used for this problem. However, existing work utilizes conventional i-vectors as the input representation and builds simple fully connected networks for metric learning, thus not fully leveraging the modeling power of DNN architectures. This paper investigates the importance of learning effective representations from the sequences directly in metric learning pipelines for speaker diarization. More specifically, we propose to employ attention models to learn embeddings and the metric jointly in an end-to-end fashion. Experiments are conducted on the CALLHOME conversational speech corpus. The diarization results demonstrate that, besides providing a unified model, the proposed approach achieves improved performance when compared against existing approaches.

A Hierarchical Attention Network-Based Approach for Depression Detection from Transcribed Clinical Interviews

“…Similar to the word-level representations, the sequential sentences representations are fused using either a naïve approach as described in Equation 4, a global attention approach as defined in Equations (8) to (11), or a contextual attention approach as detailed in Equations (15) to (17). This generates a single highlevel representationh of all sentences in the whole interview.…”

“…Recently, attention mechanisms have been employed in a broad range of applications, such as acoustic scene classification [7], speaker diarisation [8], speech emotion recognition [9], image classification [10], video classification [11], and video description [12]. Attention mechanisms with linguistic information have also been used in document classification [13] and sentiment and self-assessed emotion detection [14] problems.…”

Designing an Effective Metric Learning Pipeline for Speaker Diarization

“…Portions of this work were performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. for example cosine similarity between i−vectors, more recent solutions have emphasized the importance of integrating a metric learning pipeline into diarization systems [5,6,7]. This naturally allows knowledge inferred from an external data source to be utilized while performing diarization on an unseen target data.…”

“…This amounts to inferring key factors in data, while encoding higher order interactions, to ensure that examples from the same speaker are within smaller distances, compared to examples from a different speaker [9]. While a variety of formulations exist for supervised metric learning [7,10], recent approaches have relied on deep networks to construct embeddings that satisfy the supervisory constraints. Popular examples include the siamese [11], triplet [12,9], and quadruplet [13] networks.…”

“…Popular examples include the siamese [11], triplet [12,9], and quadruplet [13] networks. By coupling sequence modeling techniques with these deep metric learning formalisms, recent works such as [5,7] produce state-of-the-art diarization performance, while entirely dispensing the need for explicit feature design.…”