Muse: Multi-Modal Target Speaker Extraction with Visual Cues

Pan, Zexu; Tao, Ruijie; Xu, Chenglin; Li, Haizhou

doi:10.1109/icassp39728.2021.9414023

Cited by 21 publications

(9 citation statements)

References 25 publications

(49 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2) Visual encoder: The visual encoder seeks to encode the target's lip images sequence into a sequence of visual embeddings V (t) ∈ R N ×T , representing the target speaker's visemes, and in sync with the target's speech. We design the visual encoder with a structure similar to the visual encoder in MuSE [67], which consists of a 3-dimensional (3D) convolution conv3D, an 18 layer residual convolutional neural network resnet18, 5 repeated visual temporal convolutional network V -T CN , and an up-sampling layer.…”

Section: ) Speech Encodermentioning

confidence: 99%

“…The inputs to the speaker extractor are the speech embeddings X(t) and the visual embeddings V (t). The studies on the reentry model [5], TDSE [66], and MuSE [67] suggest that, by concatenating the time-aligned visual embeddings with their corresponding speech embeddings, the speaker extractor is able to effectively estimate the mask M (t). We adopt the concatenation approach at the start of the speaker extractor.…”

Section: ) Speech Encodermentioning

confidence: 99%

“…Although visemes and phonemes do not have a one-toone correspondence, visemes do provide a fine-grained cue about underlying phonetic units being spoken [64]. Therefore, visemes have been widely adopted as the auxiliary reference for speaker extraction, which performs exceptionally well to disentangle the target speaker from high overlapped speech [38], [65]- [67]. Visemes also provide a high-level cue that discriminates between speech and non-speech signals [64], thus possibly alleviating the 'false-extraction' problem.…”

mentioning

confidence: 99%

See 2 more Smart Citations

USEV: Universal Speaker Extraction With Visual Cue

Pan

2022

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

A speaker extraction algorithm seeks to extract the target speaker's speech from a multi-talker speech mixture. The prior studies focus mostly on speaker extraction from a highly overlapped multi-talker speech mixture. However, the targetinterference speaker overlapping ratios could vary over a wide range from 0% to 100% in natural speech communication, furthermore, the target speaker could be absent in the speech mixture, the speech mixtures in such universal multi-talker scenarios are described as general speech mixtures. The speaker extraction algorithm requires an auxiliary reference, such as a video recording or a pre-recorded speech, to form top-down auditory attention on the target speaker. We advocate that a visual cue, i.e., lip movement, is more informative than an audio cue, i.e., pre-recorded speech, to serve as the auxiliary reference for speaker extraction in disentangling the target speaker from a general speech mixture. In this paper, we propose a universal speaker extraction network with a visual cue, that works for all multi-talker scenarios. In addition, we propose a scenarioaware differentiated loss function for network training, to balance the network performance over different target-interference speaker pairing scenarios. The experimental results show that our proposed method outperforms various competitive baselines for general speech mixtures in terms of signal fidelity.

show abstract

Section: ) Speech Encodermentioning

confidence: 99%

Section: ) Speech Encodermentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

USEV: Universal Speaker Extraction With Visual Cue

Pan

2022

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

show abstract

“…A number of audio-visual speaker extraction works explore the visual and contextual information by using the visemephoneme mapping cues [28][29][30]. They encode the lip images into visemes using a visual encoder pre-trained on the lip reading task, in which each viseme maps to multiple phonemes.…”

Section: Introductionmentioning

confidence: 99%

VCSE: Time-Domain Visual-Contextual Speaker Extraction Network

Li¹,

Meng²,

Pan³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

Speaker extraction seeks to extract the target speech in a multitalker scenario given an auxiliary reference. Such reference can be auditory, i.e., a pre-recorded speech, visual, i.e., lip movements, or contextual, i.e., phonetic sequence. References in different modalities provide distinct and complementary information that could be fused to form top-down attention on the target speaker. Previous studies have introduced visual and contextual modalities in a single model. In this paper, we propose a two-stage time-domain visual-contextual speaker extraction network named VCSE, which incorporates visual and selfenrolled contextual cues stage by stage to take full advantage of every modality. In the first stage, we pre-extract a target speech with visual cues and estimate the underlying phonetic sequence. In the second stage, we refine the pre-extracted target speech with the self-enrolled contextual cues. Experimental results on the real-world Lip Reading Sentences 3 (LRS3) database demonstrate that our proposed VCSE network consistently outperforms other state-of-the-art baselines.

show abstract

“…The auxiliary reference can be a prerecorded reference speech signal, in which the algorithm extracts a speech signal that has a similar voice signature with the reference speech signal [13][14][15][16][17][18]. The video recording of the target speaker also serves as such a reference, in which the algorithm extracts a speech signal that is temporally synchronized with the speaker's motion in the video [19][20][21][22][23][24][25].…”

Section: Introductionmentioning

confidence: 99%

A Hybrid Continuity Loss to Reduce Over-Suppression for Time-domain Target Speaker Extraction

Pan¹,

Meng²,

Li³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Speaker extraction algorithm extracts the target speech from a mixture speech containing interference speech and background noise. The extraction process sometimes over-suppresses the extracted target speech, which not only creates artifacts during listening but also harms the performance of downstream automatic speech recognition algorithms. We propose a hybrid continuity loss function for time-domain speaker extraction algorithms to settle the over-suppression problem. On top of the waveform-level loss used for superior signal quality, i.e., SI-SDR, we introduce a multi-resolution delta spectrum loss in the frequency-domain, to ensure the continuity of an extracted speech signal, thus alleviating the oversuppression. We examine the hybrid continuity loss function using a time-domain audio-visual speaker extraction algorithm on the YouTube LRS2-BBC dataset. Experimental results show that the proposed loss function reduces the over-suppression and improves the word error rate of speech recognition on both clean and noisy two-speakers mixtures, without harming the reconstructed speech quality.

show abstract

Muse: Multi-Modal Target Speaker Extraction with Visual Cues

Cited by 21 publications

References 25 publications

USEV: Universal Speaker Extraction With Visual Cue

USEV: Universal Speaker Extraction With Visual Cue

VCSE: Time-Domain Visual-Contextual Speaker Extraction Network

A Hybrid Continuity Loss to Reduce Over-Suppression for Time-domain Target Speaker Extraction

Contact Info

Product

Resources

About