Affect Detection from Text-Based Virtual Improvisation and Emotional Gesture Recognition

Abstract

We have developed an intelligent agent to engage with users in virtual drama improvisation previously. The intelligent agent was able to perform sentence-level affect detection from user inputs with strong emotional indicators. However, we noticed that many inputs with weak or no affect indicators also contain emotional implication but were regarded as neutral expressions by the previous interpretation. In this paper, we employ latent semantic analysis to perform topic theme detection and identify target audiences for such inputs. We also discuss how such semantic interpretation of the dialog contexts is used to interpret affect more appropriately during virtual improvisation. Also, in order to build a reliable affect analyser, it is important to detect and combine weak affect indicators from other channels such as body language. Such emotional body language detection also provides a nonintrusive channel to detect users' experience without interfering with the primary task. Thus, we also make initial exploration on affect detection from several universally accepted emotional gestures.

A Survey on Emotion Recognition for Human Robot Interaction

“…For example, according to the position of hands, one can decide if an individual is straightforward (turning the hands inside towards the questioner) or deceptive (concealing the hands behind the back) [68]. Zhang and Yap (2012) [70] proposed a system for recognizing the target emotional gestures by using the Kinect sensor. Positions of the 7 joints (for example: head, right hand, left hand, right elbow, left elbow, left hip, and hip appropriate) were distinguished by utilizing OpenCV ("Open Source Computer Vision") library and distances between these extracted points were determined.…”

Studying social interactions through immersive virtual environment technology: virtues, pitfalls, and future challenges

“…Another example in this direction comes from Zhang and Yap (2012) who studied automatic affect detection based on participants’ verbal (written) and non-verbal behavior during a virtual role-play. Affect detection in verbal information was performed through latent semantic analysis, which is an algorithm that automatically learns semantic information about words through their common use in natural language ( Landauer and Dumais, 1997 ).…”

“…Moreover, we are faced with a very fast developing research domain because of the frequent technical improvements and increased availability of relatively cheap virtual reality devices which makes an update since 2009 timely. In particular, in the last years more effort has been put into integrating IVET with other technologies, such as eye-tracking ( Wieser et al, 2010 ), movement extraction devices ( Zhang and Yap, 2012 ; Batrinca et al, 2013 ), and EEG ( Kober et al, 2012 ). Moreover, recent studies have started to address the issue of making the conversation between participants and virtual humans smoother ( Malatesta et al, 2009 ; Zhang and Yap, 2012 ).…”

“…In particular, in the last years more effort has been put into integrating IVET with other technologies, such as eye-tracking ( Wieser et al, 2010 ), movement extraction devices ( Zhang and Yap, 2012 ; Batrinca et al, 2013 ), and EEG ( Kober et al, 2012 ). Moreover, recent studies have started to address the issue of making the conversation between participants and virtual humans smoother ( Malatesta et al, 2009 ; Zhang and Yap, 2012 ). In addition, more studies investigated influences on participants’ behavior, physiological responses, and cognitions either by manipulating objects in the virtual world ( Latu et al, 2013 ; Qu et al, 2013 ), avatars’ behavior ( Llobera et al, 2010 ), or participants’ physical appearance in the virtual world ( Peck et al, 2013 ).…”

Three-Dimensional, Kinematic, Human Behavioral Pattern-Based Features for Multimodal Emotion Recognition

“…Therefore, this paper uses the movement of tracked points on the face, head, hand, and body into behavioral pattern-based features or behavioral rules to represent specific emotions. The motivation to use features from behavioral patterns for emotion estimation was drawn from behavioral science research on emotional gesture recognition [18,20,21,34,35] and adaptive rule-based facial expression recognitions [19]. The studies have demonstrated unimodal affect recognition by using limited set of gesture-based rules and rules extracted from various facial expression profiles.…”