An Event Driven Fusion Approach for Enjoyment Recognition in Real-time

Abstract

Social signals and interpretation of carried information is of high importance in Human Computer Interaction. Often used for affect recognition, the cues within these signals are displayed in various modalities. Fusion of multi-modal signals is a natural and interesting way to improve automatic classification of emotions transported in social signals. Throughout most present studies, uni-modal affect recognition as well as multi-modal fusion, decisions are forced for fixed annotation segments across all modalities. In this paper, we investigate the less prevalent approach of event driven fusion, which indirectly accumulates asynchronous events in all modalities for final predictions. We present a fusion approach, handling shorttimed events in a vector space, which is of special interest for real-time applications. We compare results of segmentation based * lingenfelser@hcm-lab.de † wagner@hcm-lab.de ‡ andre@hcm-lab.de § g.mckeown@qub.ac.uk ¶ w.curran@qub.ac.uk uni-modal classification and fusion schemes to the event driven fusion approach. The evaluation is carried out via detection of enjoyment-episodes within the audiovisual Belfast Story-Telling Corpus.

KRISTINA: A Knowledge-Based Virtual Conversation Agent

“…Extracted facial and paralinguistic cues are combined through fusion strategies in order to generate a final prediction. Our work on fusion draws on Lingenfelser's [20] "event-driven" fusion, which is based on [13]. The algorithm does not force decisions throughout considered modalities for each time frame, but instead asynchronously fuses time-sensitive events from any given number of modi.…”

“…This has the advantage of incorporating temporal alignments between modi and being very flexible with respect to the type and mode of used events. In [20], this algorithm was used to combine the recognition of short-timed laugh (audio) and smile (video) events for a continuous assessment of a user's level of positive valence. For KRISTINA, it is extended to cover the whole valence arousal space, spanned by positive and negative valence and arousal axes.…”

Laughter Research: A Review of the ILHAIRE Project

“…In [63], an event driven real-time fusion system was proposed. It rather corresponds to a late fusion mechanism, with some additional time-based accumulation.…”

The Belfast storytelling database: A spontaneous social interaction database with laughter focused annotation

“…This decision determines if the observed window is labeled as a laughter window. A detailed description of the recognition system is found in [23]. Once trained on a subset of the corpus, it can be used to generate automated annotations for the remaining sessions.…”