Facial expression GAN for voice-driven face generation

Zheng, Fang; Liu, Zhen; Liu, Tingting; Hung, Chih‐Chieh; Xiao, Jiangjian; Feng, Guangjin

doi:10.1007/s00371-021-02074-w

Cited by 21 publications

(5 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, Wen et al [228] develops a voice embedding network consisting of six convolution layers to learn speech features. A voice encoder included voice activity detection and V-net is used in Fang et al [250] to output audio embedding.…”

Section: Speech-to-face Generationmentioning

confidence: 99%

“…For better identity matching, Wen et al [228] introduced the second discriminator to verify the identity of face image output. Considering that emotional expression is a key face attribute of a realistic face image, Fang et al [250] applied two classifiers to measure identity and emotion semantic relevance in generating. In [253], a Face-based Residual Personalized Speech Synthesis Model (FR-PSS) containing a speech encoder, a speech synthesizer and a face encoder is designed for PSS.…”

Section: Speech-to-face Generationmentioning

confidence: 99%

“…Identity preservation LDM [246], Cosine, L 2 , L 1 [246], Face retrieval [246], Identity recognition [228], Gender classification [228] Image quality IS [250], FID [250] 5.4.2 Talking Head Generation Given a target face image and a speech clip, TH generation aims at synthesizing a sequence of target face images where the lip motion, head pose, and facial expressions are synchronized with the audio. Significantly different from the speech-to-face generation task, which extracts the identity of the speaker from the given speech, the TH generation task focuses on the content of the speech.…”

Section: Metrics' Degree Metricsmentioning

confidence: 99%

“…Spectrogram, MFCC, and Fbank are the three mostly used audio features. Fang et al [250] performs an ablation experiment on these three audio features, and they found that Fbank achieved the best performance, while the Spectrogram performed the worst FID. The reasons they guessed that Spectrogram contained much redundant information, MFCC discarded some related information, and Fbank kept balance.…”

Section: Multi-modal Feature Extractionmentioning

confidence: 99%

See 3 more Smart Citations

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Liu

Wang

et al. 2021

Front. Neurorobot.

View full text Add to dashboard Cite

The redundant information, noise data generated in the process of single-modal feature extraction, and traditional learning algorithms are difficult to obtain ideal recognition performance. A multi-modal fusion emotion recognition method for speech expressions based on deep learning is proposed. Firstly, the corresponding feature extraction methods are set up for different single modalities. Among them, the voice uses the convolutional neural network-long and short term memory (CNN-LSTM) network, and the facial expression in the video uses the Inception-Res Net-v2 network to extract the feature data. Then, long and short term memory (LSTM) is used to capture the correlation between different modalities and within the modalities. After the feature selection process of the chi-square test, the single modalities are spliced to obtain a unified fusion feature. Finally, the fusion data features output by LSTM are used as the input of the classifier LIBSVM to realize the final emotion recognition. The experimental results show that the recognition accuracy of the proposed method on the MOSI and MELD datasets are 87.56 and 90.06%, respectively, which are better than other comparison methods. It has laid a certain theoretical foundation for the application of multimodal fusion in emotion recognition.

show abstract

Section: Speech-to-face Generationmentioning

confidence: 99%

Section: Speech-to-face Generationmentioning

confidence: 99%

Section: Metrics' Degree Metricsmentioning

confidence: 99%

Section: Multi-modal Feature Extractionmentioning

confidence: 99%

See 2 more Smart Citations

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Liu

Wang

et al. 2021

Front. Neurorobot.

View full text Add to dashboard Cite

show abstract

“…Shi et al [24] designed a GAN that transforms 2D human face images into 3D images. Fang et al [25] developed a GAN to produce human faces from human speech fragments. Chen et al [26] proposed a GAN that uses two discriminators at the same time to repair images.…”

Section: Introductionmentioning

confidence: 99%

Application of Generative Adversarial Network and Diverse Feature Extraction Methods to Enhance Classification Accuracy of Tool-Wear Status

Chen

Loh

et al. 2022

Electronics

View full text Add to dashboard Cite

The means of accurately determining tool-wear status has long been important to manufacturers. Tool-wear status classification enables factories to avoid the unnecessary costs incurred by replacing tools too early and to prevent product damage caused by overly worn tools. While researchers have examined this topic for over a decade, most existing studies have focused on model development but have neglected two fundamental issues in machine learning: data imbalance and feature extraction. In view of this, we propose two improvements: (1) using a generative adversarial network to generate realistic computer numerical control machine vibration data to overcome data imbalance and (2) extracting features in the time domain, the frequency domain, and the time–frequency domain simultaneously for modeling and integrating these in an ensemble model. The experiment results demonstrate how both proposed modifications are reasonable and valid.

show abstract

Beyond view transformation: feature distribution consistent GANs for cross-view gait recognition

2021

View full text Add to dashboard Cite

Facial expression GAN for voice-driven face generation

Cited by 21 publications

References 27 publications

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Application of Generative Adversarial Network and Diverse Feature Extraction Methods to Enhance Classification Accuracy of Tool-Wear Status

Beyond view transformation: feature distribution consistent GANs for cross-view gait recognition

Contact Info

Product

Resources

About