Supervised Contrastive Learning for Affect Modelling

Abstract

Figure 1: A high-level visualisation of the concept introduced. Supervised contrastive learning operates by infusing affect information within the representation, by pairing positive embeddings and dissociating negative embeddings. We assume affect is embedded in the multimodal latent space and defines what distinguishes (contrasts) data. Positive (green) and negative (red) multimodal data is labelled with respect to an anchor affect (black). Similar and dissimilar affect patterns define, respectively, positive and negative pairs of the anchor. The resulting representation is general with respect to affect patterns in a corpus.

A Graph Neural Network for EEG-Based Emotion Recognition With Contrastive Learning and Generative Adversarial Neural Network Data Augmentation

“…CL is a self-supervised learning method to project the data into a space where different views of the same input sample have highly similar representations. While initially developed for image classification [54,58] and computer vision applications [59], CL has also been applied to emotion recognition based on physiological signals [60][61][62]. Mohsenvand et al [60] used a similar method to SimCLR, which they called SeqCLR, to learn similarities between differently augmented transforms of the same EEG data sample, disregarding the emotional state of the data sample.…”

“…Pinitas et al [61] proposed a model to learn general affectinfused multi-modal representations from audio, video, Electrocardiography (ECG), and Electrodermal Activity (EDA) modalities. The model was built upon the contrastive learning framework introduced in [58].…”

Predicting Player Engagement in Tom Clancy's The Division 2: A Multimodal Approach via Pixels and Gamepad Actions

“…Zhang et al [59] used a Convolutional LSTM and a 1D-CNN to extract spatio-temporal facial and bio-sensing features, respectively. Recently, Pinitas et al [43] employed Supervised Contrastive Learning on audiovisual and physiological data to model arousal. Unlike the aforementioned studies, this paper presents preliminary findings regarding longterm player engagement prediction from in-game footage and game controller input using CNNs.…”

“…As this study aims to model long-term engagement via players' multimodal signals, we consider the following data pre-processing method. We split each participant's session (video) into overlapping time windows [29,43] using a sliding step of 1.5 seconds and a window length of 10 seconds, corresponding to 22, 541 samples in the entire clean dataset. The sliding step and window length are essential hyperparameters since they influence, respectively, the size of the dataset and the information contained in each window.…”

“…When analysing an entirely new affect corpus such as The Division 2, it is useful for all possible learning paradigms to be explored-including regression, classification and ordinal learning [55]. In this initial study of The Division 2, we focus on affect classification since it is one of the most commonly used learning paradigms in player and affect modelling [26][27][28][29]43].…”