Modeling continuous self-report measures of perceived emotion using generalized additive mixed models.

Abstract

Emotion research has long been dominated by the "standard method" of displaying posed or acted static images of facial expressions of emotion. While this method has been useful, it is unable to investigate the dynamic nature of emotion expression. Although continuous self-report traces have enabled the measurement of dynamic expressions of emotion, a consensus has not been reached on the correct statistical techniques that permit inferences to be made with such measures. We propose generalized additive models and generalized additive mixed models as techniques that can account for the dynamic nature of such continuous measures. These models allow us to hold constant shared components of responses that are due to perceived emotion across time, while enabling inference concerning linear differences between groups. The generalized additive mixed model approach is preferred, as it can account for autocorrelation in time series data and allows emotion decoding participants to be modeled as random effects. To increase confidence in linear differences, we assess the methods that address interactions between categorical variables and dynamic changes over time. In addition, we provide comments on the use of generalized additive models to assess the effect size of shared perceived emotion and discuss sample sizes. Finally, we address additional uses, the inference of feature detection, continuous variable interactions, and measurement of ambiguity.

On the Influence of an Iterative Affect Annotation Approach on Inter-Observer and Self-Observer Reliability

“…One solution is to compensate for the inherent ambiguity in affect annotation using multilabel classification techniques (not to be confused with multi-class classification [75]) where multiple viewpoints on the same affective state can co-exist. Another is to attempt to increase reliability and the simplest way to do this is to increase the number of observers [9,52]. Or perhaps, it might be advisable to consider both options.…”

“…The basic intuition is that the easiest way to increase reliability is to simply increase the number of observers so that observer-specific variance gets cancelled out. McKeown and Sneddon [52] performed a simulated analysis and concluded that 20-30 observers were needed to confidently detect an emotional signal from time-continuous annotations. Although they are careful to make no claims on generalizability of this result to other situations 2 , requiring such a large number of observers might not always be feasible.…”

“…We provided multiple reasons for the low SO reliability (see Sections 1.2, 1.5, and 4.1), but it would be prudent to increase OO reliability in order to more confidently address the issue of low SO reliability. As noted above, both Rosenthal [9] and McKeown and Sneddon [52] provide guidelines for selection of an appropriate number of observers needed to achieve a desired level of effective reliability. Therefore, only after increasing OO reliability (by increasing the number of observers) can we make more definitive statements on the status of SO reliability.…”

“…Figure 6 provides a visualization of the nonlinear relationship between OO kappa and iteration. It was produced using a generalized additive mixed model (GAMM) approach [52], which consists of modeling a response variable via an additive combination of parametric and nonparametric smooth functions of predictor variables. GAMMs are an extension of standard generalized additive models (GAMs) by modeling autocorrelations among residuals (since the data are time series).…”

Context-Aware Automated Analysis and Annotation of Social Human--Agent Interactions

“…According to McKeown and Sneddon [2014], the standard method for annotating data is to have fixed and known time segments associated with a descriptive label. In our system, agent events deliver start and end boundaries automatically, as they are triggered by the interaction modeling component.…”