DyadGAN: Generating Facial Expressions in Dyadic Interactions

Huang, Yuchi; Khan, Saad

doi:10.1109/cvprw.2017.280

Cited by 60 publications

(31 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…One similar work that uses a probabilistic method is DyadGAN [25], which trained a conditional GAN to generate face images based on the interlocutor's facial expressions. However, the work only produced a single image, ignoring temporal aspects.…”

Section: Interlocutor-aware Gesture Generationmentioning

confidence: 99%

“…As modeling conversational dynamics is difficult to achieve, most non-verbal behavior generation methods only use speech and/or semantic content produced by the agent as inputs to the system [27,32,45]. Recently a few systems have been introduced that use non-verbal behaviors from the interlocutor to control non-verbal output from the system [1,17,20,25]. We continue this line of work and present a probabilistic system, based on normalizing flows, for generating facial gestures in dyadic settings.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Let's Face It

Jonell

Kucherenko

Henter

et al. 2020

Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

View full text Add to dashboard Cite

To enable more natural face-to-face interactions, conversational agents need to adapt their behavior to their interlocutors. One key aspect of this is generation of appropriate non-verbal behavior for the agent, for example facial gestures, here defined as facial expressions and head movements. Most existing gesture-generating systems do not utilize multi-modal cues from the interlocutor when synthesizing non-verbal behavior. Those that do, typically use deterministic methods that risk producing repetitive and non-vivid motions. In this paper, we introduce a probabilistic method to synthesize interlocutor-aware facial gestures ś represented by highly expressive FLAME parameters ś in dyadic conversations. Our contributions are: a) a method for feature extraction from multi-party video and speech recordings, resulting in a representation that allows for independent control and manipulation of expression and speech articulation in a 3D avatar; b) an extension to MoGlow, a recent motion-synthesis method based on normalizing flows, to also take multi-modal signals from the interlocutor as input and subsequently output interlocutor-aware facial gestures; and c) a subjective evaluation assessing the use and relative importance of the different modalities in the synthesized output. The results show that the model successfully leverages the input from the interlocutor to generate more appropriate behavior. Videos, data, and code are available at: https://jonepatr.github.io/lets_face_it/.

show abstract

Section: Interlocutor-aware Gesture Generationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Let's Face It

Jonell

Kucherenko

Henter

et al. 2020

Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

View full text Add to dashboard Cite

show abstract

“…Additionally, a range of metric-based approaches have been proposed to quantitatively evaluate the adversarial training frameworks. For example, the authors in [51] compared the intra-set and inter-set average Euclidean distances between different sets of the generated faces. Similarly, to quantitatively evaluate models for emotion conversion, other evaluation measurements raised in the literature include BiLingual Evaluation Understudy (BLEU) [52] and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [53] for text, and a signal-to-noise ratio test for speech [15].…”

Section: Performance Evaluationmentioning

confidence: 99%

“…Radfod et al [19] 2016 SYN image DCGAN vector arithmetic can be done in latent vector space, e. g., smiling woman -neutral woman + neutral man = smiling man Chen et al [69] 2016 SYN image infoGAN latent code can be interpreted; support gradual transformation Huang & Khan [51] 2017 SYN image/video DyadGAN interaction scenario; identity + attribute from the interviewee Melis & Amores [83] 2017 SYN Image (art) cGAN generate emotional artwork Bao et al [49] 2018 SYN image identity preserving GAN the identity and attributes of faces are separated Pham et al [81] 2018 SYN image/video GATH conditioned by AU; from static image to dynamic video Song et al [82] 2018 SYN image/video cGAN static image to dynamic video, conditioned by audio sequences Nojavansghari et al [50] 2018 SYN image DyadGAN (with RNN) and sentiment analysis has yet to be reported, to answer questions such as which model is faster, more accurate, or easier to implement. This absence is mainly due to the lack of benchmark datasets and thoughtfully designed metrics for each specific application (i. e., emotion generation, emotion conversion, and emotion perception and understanding).…”

Section: B Other Ongoing Breakthroughsmentioning

confidence: 99%

Adversarial Training in Affective Computing and Sentiment Analysis: Recent Advances and Perspectives [Review Article]

Han

Zhang

Schuller

2019

IEEE Comput. Intell. Mag.

View full text Add to dashboard Cite

Over the past few years, adversarial training has become an extremely active research topic and has been successfully applied to various Artificial Intelligence (AI) domains. As a potentially crucial technique for the development of the next generation of emotional AI systems, we herein provide a comprehensive overview of the application of adversarial training to affective computing and sentiment analysis. Various representative adversarial training algorithms are explained and discussed accordingly, aimed at tackling diverse challenges associated with emotional AI systems. Further, we highlight a range of potential future research directions. We expect that this overview will help facilitate the development of adversarial training for affective computing and sentiment analysis in both the academic and industrial communities.

show abstract

“…Bavelas [4] clearly distinguished the different functions of feedback and defined two types of feedback: generic and specific. Generic feedback simply signals that the listener is [12] A:"Something like …" [11] C:"How about Sushi …" [10] B:"What do you …" [13] C:"But, this may…" [14] B: "That's right"…”

Section: Introductionmentioning

confidence: 99%

Estimating the Intensity of Facial Expressions Accompanying Feedback Responses in Multiparty Video-Mediated Communication

Ueno

Nakano

Zeng

et al. 2020

Proceedings of the 2020 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Providing feedback to a speaker is an essential communication signal for maintaining a conversation. In specific feedback, which indicates the listener's reaction to the speaker's utterances, the facial expression is an effective modality for conveying the listener's reactions. Moreover, not only the type of facial expressions, but also the degree of intensity of the expressions, may influence the meaning of the specific feedback. In this study, we propose a multimodal deep neural network model that predicts the intensity of facial expressions co-occurring with feedback responses. We focus on multiparty video-mediated communication. In video-mediated communication, close-up frontal face images of each participant are continuously presented on the display; the attention of the participants is more likely to be drawn to the facial expressions. We assume that in such communication, the importance of facial expression in the listeners' feedback responses increases. We collected 33 video-mediated conversations by groups of three people and obtained audio and speech data for each participant. Using the corpus collected as a dataset, we created a deep neural network model that predicts the intensity of 17 types of action units (AUs) co-occurring with the feedback responses. The proposed method employed GRU-based model with attention mechanism for audio, visual, and language modalities. A decoder was trained to produce the intensity values for the 17 AUs frame by frame. In the experiment, unimodal and multimodal models were compared in terms of their performance in predicting salient AUs that characterize facial expression in feedback responses. The results suggest that wellperforming models differ depending on the AU categories; audio information was useful for predicting AUs that express happiness,

show abstract

DyadGAN: Generating Facial Expressions in Dyadic Interactions

Cited by 60 publications

References 16 publications

Let's Face It

Let's Face It

Adversarial Training in Affective Computing and Sentiment Analysis: Recent Advances and Perspectives [Review Article]

Estimating the Intensity of Facial Expressions Accompanying Feedback Responses in Multiparty Video-Mediated Communication

Contact Info

Product

Resources

About