Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition

Abstract

This paper proposes to recognise the true (self-reported) personality from the learned simulation of the target subject's cognition. This approach builds on two following findings in cognitive science: (i) human cognition partially determines expressed behaviour and is directly linked to true personality traits; and (ii) in dyadic interactions individuals' nonverbal behaviours are influenced by their conversational partner's behaviours. In this context, we hypothesise that during a dyadic interaction, a target subject's facial reactions are driven by two main factors, i.e. their internal (person-specific) cognitive process, and the externalised nonverbal behaviours of their conversational partner. Consequently, we propose to represent the target subject's (defined as the listener) person-specific cognition in the form of a person-specific CNN architecture that has unique architectural parameters and depth, which takes audio-visual non-verbal cues displayed by the conversational partner (defined as the speaker) as input, and is able to reproduce the target subject's facial reactions. Each personspecific CNN is explored by the Neural Architecture Search (NAS) and a novel adaptive loss function, which is then represented as a graph representation for recognising the target subject's true personality. Experimental results not only show that the produced graph representations are well associated with target subjects' personality traits in both human-human and human-machine interaction scenarios, and outperform the existing approaches with significant advantages, but also demonstrate that the proposed novel strategies such as adaptive loss, and the end-to-end vertices/edges feature learning, help the proposed approach in learning more reliable personality representations. Building on our earlier version of this work, this paper further proposes: (i) assigning a unique depth for each CNN; (ii) a novel end-to-end graph vertex feature learning strategy; (iii) a transformer-based edge feature learning strategy; and (iv) evaluating the approach in human-machine interaction scenario.

Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition

“…We totally train the proposed model for 40 epochs, including Evaluation Metric. We follow previous AU occurrence recognition studies [Shao et al, 2021a;Churamani et al, 2021;Li et al, 2019b;Song et al, 2021c] using a common metric: frame-based F1 score, to evaluate the performance of our approach, which is denoted as F 1 = 2 P •R P +R . It takes the recognition precision P and recall rate R into consideration.…”

“…As a result, [Li et al, 2018] 39.0 35.2 48.6 76.1 72.9 81.9 86.2 58.8 37.5 59.1 35.9 35.8 55.9 JAA-Net [Shao et al, 2018] 47.2 44.0 54.9 77.5 74.6 84.0 86.9 61.9 43.6 60.3 42.7 41.9 60.0 LP-Net [Niu et al, 2019] 43.4 38.0 54.2 77.1 76.7 83.8 87.2 63.3 45.3 60.5 48.1 54.2 61.0 ARL [Shao et al, 2019] 45.8 39.8 55.1 75.7 77.2 82.3 86.6 58.8 47.6 62.1 47.4 [55.4] 61.1 SEV-Net [Yang et al, 2021] [58.2] [50.4] 58.3 [81.9] 73.9 [87.8] 87.5 61.6 [52.6] 62.2 44.6 47.6 63.9 FAUDT [Jacob and Stenger, 2021] 51.7 [49.3] [61.0] 77.8 79.5 82.9 86.3 [67.6] 51.9 63.0 43.7 [56.3] 64.2 SRERL [Li et al, 2019a] 46.9 45. [Li et al, 2018] 41.5 26.4 66.4 50.7 [80.5] [89.3] 88.9 15.6 48.5 JAA-Net [Shao et al, 2018] 43.7 46.2 56.0 41.4 44.7 69.6 88.3 58.4 56.0 LP-Net [Niu et al, 2019] 29.9 24.7 72.7 46.8 49.6 72.9 93.8 65.0 56.9 ARL [Shao et al, 2019] 43.9 the categorical cross-entropy loss is introduced as:…”

“…activated simultaneously. While previous studies found that underlying relationships among AUs' activation [Corneanu et al, 2018;Song et al, 2021c;Shao et al, 2021a] are crucial for their recognition, how to properly model such relationships is still an open research question in the field.…”

“…A popular strategy applies various traditional machine learning models (e.g., conditional models [Eleftheriadis et al, 2015]) or neural network-based operations (e.g., convolution [Zhao et al, 2016], Long-Short-Term-Memory networks [Niu et al, 2019] or attention [Shao et al, 2021a]), to encode all AU descriptors as a single representation which reflects the underlying relationship among all AUs. A key drawback of such solutions is that they fail to individually model the relationship between each pair of AUs, which may contain crucial cues for their recognition (Problem 1).…”

“…Our multi-dimensional edge encodes unique and multiple relationships between each pair of AUs, rather than a single relationship (e.g., spatial adjacency, co-occurrence patterns, etc.) that the single value-edge encoded, which would theoretically generalizes better in modeling complex relationships between vertices [Gong and Cheng, 2019;Song et al, 2021a;Shao et al, 2021b]. The main novelty of the proposed approach in comparison to pre-defined AU graphs [Li et al, 2019a;Liu et al, 2020] and deep learned facial displayspecific graphs [Song et al, 2021b;Song et al, 2021c] are illustrated in Figure 1.…”