Speaker Extraction With Co-Speech Gestures Cue

Pan, Zexu; Qian, Xinyuan; Li, Haizhou

doi:10.1109/lsp.2022.3175130

Cited by 8 publications

(4 citation statements)

References 46 publications

(38 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Text2sign [21] HLSTM [22] ESN [23] DiffGAN [24] RG [25] SEEG [26] HA2G [27] Paul Duchnowski [28] Gérard Bailly [29] Wav2lip [30] Audio2head [31] AD-NERF [32] DiffTalk [33] Rule-based [34] LBP [35] SDF [36] PTSLP [37] Syn [38] MMFSL [39] Re-Syn [40] CMML [41] DTW [42] HMMs [43] FCN [44] RL [45] CS [1] MCCS [46] SLreview [2] UnorgSign [47] CoSreview [6] CoSreview [3] SE [48] THreview [4] THE [49] VHTHG [50] Fig. 3: Structured taxonomy of the existing BL research which includes three genres.…”

Section: Sign Language Recognitionmentioning

confidence: 99%

“…CoS signals not only play a crucial role in enhancing the clarity, expressiveness and emotional content of verbal communication, but also capture the rich communicative context, and reveal the speaker's social identity and cultural affiliation [48]. Therefore, it is a growing trend towards exploring multi-modal approaches that take into account both the visual information from gestures and the accompanying speech signals, which allows for more comprehensive and accurate analysis in the areas such as emotion recognition and dialogue understanding [75].…”

Section: Co-speechmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Liu

Wang

et al. 2021

Front. Neurorobot.

View full text Add to dashboard Cite

The redundant information, noise data generated in the process of single-modal feature extraction, and traditional learning algorithms are difficult to obtain ideal recognition performance. A multi-modal fusion emotion recognition method for speech expressions based on deep learning is proposed. Firstly, the corresponding feature extraction methods are set up for different single modalities. Among them, the voice uses the convolutional neural network-long and short term memory (CNN-LSTM) network, and the facial expression in the video uses the Inception-Res Net-v2 network to extract the feature data. Then, long and short term memory (LSTM) is used to capture the correlation between different modalities and within the modalities. After the feature selection process of the chi-square test, the single modalities are spliced to obtain a unified fusion feature. Finally, the fusion data features output by LSTM are used as the input of the classifier LIBSVM to realize the final emotion recognition. The experimental results show that the recognition accuracy of the proposed method on the MOSI and MELD datasets are 87.56 and 90.06%, respectively, which are better than other comparison methods. It has laid a certain theoretical foundation for the application of multimodal fusion in emotion recognition.

show abstract

Section: Sign Language Recognitionmentioning

confidence: 99%

Section: Co-speechmentioning

confidence: 99%

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Liu

Wang

et al. 2021

Front. Neurorobot.

View full text Add to dashboard Cite

show abstract

“…Although a great success, there is inherent ambiguity in speaker labeling of the separated signals. An auxiliary reference, such as a pre-recorded speech signal [13][14][15][16] or video frame sequence [17][18][19][20], can be used to solve the speaker ambiguity. The speaker extraction algorithm employs such auxiliary reference to form top-down attention on the target speaker and extracts its speech.…”

Section: Introductionmentioning

confidence: 99%

VCSE: Time-Domain Visual-Contextual Speaker Extraction Network

Li¹,

Meng²,

Pan³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

Speaker extraction seeks to extract the target speech in a multitalker scenario given an auxiliary reference. Such reference can be auditory, i.e., a pre-recorded speech, visual, i.e., lip movements, or contextual, i.e., phonetic sequence. References in different modalities provide distinct and complementary information that could be fused to form top-down attention on the target speaker. Previous studies have introduced visual and contextual modalities in a single model. In this paper, we propose a two-stage time-domain visual-contextual speaker extraction network named VCSE, which incorporates visual and selfenrolled contextual cues stage by stage to take full advantage of every modality. In the first stage, we pre-extract a target speech with visual cues and estimate the underlying phonetic sequence. In the second stage, we refine the pre-extracted target speech with the self-enrolled contextual cues. Experimental results on the real-world Lip Reading Sentences 3 (LRS3) database demonstrate that our proposed VCSE network consistently outperforms other state-of-the-art baselines.

show abstract

“…A recent work [23] uses audio features to select relevant visual features in combination with an attention mechanism and a data augmentation strategy in order to address low-resolution, lip-occlusion, and outof-sync problems altogether. The SEG network [24] avoids the need for lip recordings since it utilizes the co-speech gesture cue from the upper-body video recording as a reference, which is less prone to occlusions.…”

Section: Introductionmentioning

confidence: 99%

ImagineNET: Target Speaker Extraction with Intermittent Visual Cue through Embedding Inpainting

Pan¹,

Wang²,

Borsdorf³

et al. 2022

Preprint

View full text Add to dashboard Cite

The speaker extraction technique seeks to single out the voice of a target speaker from the interfering voices in a speech mixture. Typically an auxiliary reference of the target speaker is used to form voluntary attention. Either a pre-recorded utterance or a synchronized lip movement in a video clip can serve as the auxiliary reference. The use of visual cue is not only feasible, but also effective due to its noise robustness, and becoming popular. However, it is difficult to guarantee that such parallel visual cue is always available in real-world applications where visual occlusion or intermittent communication can occur. In this paper, we study the audiovisual speaker extraction algorithms with intermittent visual cue. We propose a joint speaker extraction and visual embedding inpainting framework to explore the mutual benefits. To encourage the interaction between the two tasks, they are performed alternately with an interlacing structure and optimized jointly. We also propose two types of visual inpainting losses and study our proposed method with two types of popularly used visual embeddings. The experimental results show that we outperform the baseline in terms of signal quality, perceptual quality, and intelligibility.

show abstract

Speaker Extraction With Co-Speech Gestures Cue

Cited by 8 publications

References 46 publications

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

VCSE: Time-Domain Visual-Contextual Speaker Extraction Network

ImagineNET: Target Speaker Extraction with Intermittent Visual Cue through Embedding Inpainting

Contact Info

Product

Resources

About