PAD-based multimodal affective fusion

Abstract

The study of multimodality is comparatively less developed for Affective interfaces than for their traditional counterparts. However, one condition for the successful development of Affective interface technologies is the development of frameworks for the real-time multimodal fusion. In this paper, we describe an approach to multimodal affective fusion, which relies on a dimensional model, PleasureArousal-Dominance (PAD) to support the fusion of affective modalities, each input modality being represented as a PAD vector. We describe how this model supports both affective content fusion and temporal fusion within a unified approach. We report results from early user studies which confirm the existence of a correlation between measured affective input and user temperament scores.

Evaluating multimodal affective fusion using physiological signals

“…The aggregate user affective experience is then characterised by the trajectory of this vector in 3-D PAD space, over the length of an interactive session ( Figure 1d). This is described in more detail in [12].…”

“…We have previously described a system to study how multimodal affective interfaces could be used to capture user experience, using a variety of unobtrusive channels such as video and speech [12]. Input modalities consist of both user attitudes (bodily movements and posture [17,22], as well as more traditional non-verbal behaviour [14]), and affective content of speech utterances (considering both acoustic parameters and affective interpretation of specific keywords).…”

“…However, this is dependent on an appropriate underlying emotional model as a unifying representation across modalities, and we have suggested that dimensional emotion models in particular could be at the centre of the fusion mechanism [12].…”

“…In this paper, we evaluate a previously described multimodal affective fusion approach, described in [12], based on a dimensional model (Pleasure-ArousalDominance or PAD [28]), by using physiological signals which have been demonstrated to correlate strongly with the individual dimensions of this model. Our aim is to show that physiological signals associated with dimensions of Valence (Pleasure) or Arousal are consistent with the output of a PAD-based affective fusion mechanism.…”

An Event Driven Fusion Approach for Enjoyment Recognition in Real-time

“…The proposed fusion algorithm is based on preceding work done by [8] (section 2.3). We generalize this approach by designing a fusion scheme that operates in a userdefined vector space.…”