Analysis of emotion recognition using facial expressions, speech and multimodal information

Emotion Recognition System by a Neural Network Based Facial Expression Analysis

“…Systems that try to integrate both senses creating thereby multimodal systems also exist [9,10]. The visual systems can be further divided into those that analyze video sequences and those that analyze only static images.…”

“…Busso et al [9] have merged emotion recognition via video and voice in two ways. In the first method, audio and visual features were led to a single classifier, while in the second, auditory and visual information were processed in separate classifiers and then adequately combined.…”

“…Every action unit is linked to a certain facial muscle and its movement that is visible on the face. This coding scheme was an inspiration to many studies that used images and video for automatic tracing of facial features for categorizing different facial expressions [9,[12][13][14]. An example of that coding can be seen in Figure 1 which displays types of face movement monitoring (Figure 1a) by utilizing a wire frame model (Figure 1b) gained from the PBVD face tracker [12].…”

“…Methods used for analysis of video information usually differentiate themselves by groups of characteristics or features taken into account or by the classifier used, as described in [9,10,12,[14][15][16]. There are various types of classifiers that mainly depend on the technology used, from hidden Markov models, neural networks, fuzzy systems to genetic algorithms.…”

“…Therefore, in order to discern movement of points on the face it is necessary to set certain facial reference points or lines which need to have fixed positions or values in order to serve as landmarks for facial movement. Tip of the nose is commonly used as a reference point since the nose does not participate in facial expressions or mimics [9].…”