On the Influence of an Iterative Affect Annotation Approach on Inter-Observer and Self-Observer Reliability

Abstract

Affect detection systems require reliable methods to annotate affective data. Typically, two or more observers independently annotate audio-visual affective data. This approach results in inter-observer reliabilities that can be categorized as fair (Cohen's kappas of approximately .40). In an alternative iterative approach, observers independently annotate small amounts of data, discuss their annotations, and annotate a different sample of data. After a pre-determined reliability threshold is reached, the observers independently annotate the remainder of the data. The effectiveness of the iterative approach was tested in an annotation study where pairs of observers annotated affective video data in nine annotate-discuss iterations. Self-annotations were previously collected on the same data. Mixed effects linear regression models indicated that inter-observer agreement increased (unstandardized coefficient B = .031) across iterations, with agreement in the final iteration reflecting a 64% improvement over the first iteration. Follow-up analyses indicated that the improvement was nonlinear in that most of the improvement occurred after the first three iterations (B = .043), after which agreement plateaued (B ≈ 0). There was no notable complementary improvement (B ≈ 0) in self-observer agreement, which was considerably lower than observer-observer agreement. Strengths, limitations, and applications of the iterative affective annotation approach are discussed.

The Future Belongs to the Curious: Towards Automatic Understanding and Recognition of Curiosity in Children

“…Agreement level between 0.4 and 0.6 shows moderate agreement and values between 0.6 and 0.8 show substantial agreement level between raters [23]. These agreement levels compare favorably to previous work in affective computing [24]. The annotations are provided on a Likert scale from 1-4 for all emotion labels except valence, which is annotated on a scale from 1-7; representing strongly negative to strongly positive.…”

EXTRACTOR: Extracting Attack Behavior from Threat Reports

“…Then, on several discussions with the annotators, we discuss and clarify the notion of the attack behavior (i.e., productive) versus the rest (i.e., non-productive) to ensure the understanding of attack behavior is accurate. Following prior guidelines and studies (i.e., [33] and [30]), the annotation task begins in an iterative fashion. In each round, 200 messages are assigned, and disagreements are discussed with each annotator.…”

K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations

“…This annotation method we employed, a retrospective affect judgment protocol , is widely used in affective computing to collect self-reports of emotions, especially in studies where an uninterrupted engagement of subjects during an emotion induction process is essential 76 – 79 . Likewise, we opted for this method as participants’ natural interaction was necessary for acquiring quality emotion data.…”