CLIFER: Continual Learning with Imagination for Facial Expression Recognition

Churamani, Nikhil; Güneş, Hatice

doi:10.1109/fg47880.2020.00110

Cited by 25 publications

(36 citation statements)

References 33 publications

(61 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This significantly reduces the number of parameters in the decoder (≈ 2.8M as opposed to 10M in [31]) while also improving the photo-realistic quality of the reconstructed images. The decoder, adapted from the generator model of [5], implements 4 stacked conv layers, using ReLU activation, performing transposed convolutions with 128, 64, 32, 16 filters, respectively, of size (5×5) each with a stride of (2×2). The output of the last conv layer is passed through another transposed conv layer using tanh activation to generate the resultant image (x gen ) with the same dimensions as the FoI (x r ).…”

Section: Decoder For Image Reconstructionmentioning

confidence: 99%

Spatio-Temporal Analysis of Facial Actions using Lifecycle-Aware Capsule Networks

Churamani¹,

Kalkan²,

Güneş³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Most state-of-the-art approaches for Facial Action Unit (AU) detection rely upon evaluating facial expressions from static frames, encoding a snapshot of heightened facial activity. In real-world interactions, however, facial expressions are usually more subtle and evolve in a temporal manner requiring AU detection models to learn spatial as well as temporal information. In this paper, we focus on both spatial and spatio-temporal features encoding the temporal evolution of facial AU activation. For this purpose, we propose the Action Unit Lifecycle-Aware Capsule Network (AULA-Caps) that performs AU detection using both frame and sequence-level features. While at the frame-level the capsule layers of AULA-Caps learn spatial feature primitives to determine AU activations, at the sequence-level, it learns temporal dependencies between contiguous frames by focusing on relevant spatio-temporal segments in the sequence. The learnt feature capsules are routed together such that the model learns to selectively focus more on spatial or spatio-temporal information depending upon the AU lifecycle. The proposed model is evaluated on the commonly used BP4D and GFT benchmark datasets obtaining stateof-the-art results on both the datasets.

show abstract

Section: Decoder For Image Reconstructionmentioning

confidence: 99%

Spatio-Temporal Analysis of Facial Actions using Lifecycle-Aware Capsule Networks

Churamani¹,

Kalkan²,

Güneş³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…On the other hand, the emotion recognition module does not need to be constantly modified and is trained only once. While research has shown that personalizing emotion recognition in the context of continual learning increases performance [48,49], the same can be argued for action recognition (personalizing) [50,51]. In this work, we focus on IL in the context of allowing the addition of new classes to the system-personalized adaptation is out of our scope.…”

Section: Methodsmentioning

confidence: 98%

Visual Robotic Perception System with Incremental Learning for Child–Robot Interaction Scenarios

et al. 2021

View full text Add to dashboard Cite

This paper proposes a novel lightweight visual perception system with Incremental Learning (IL), tailored to child–robot interaction scenarios. Specifically, this encompasses both an action and emotion recognition module, with the former wrapped around an IL system, allowing novel actions to be easily added. This IL system enables the tutor aspiring to use robotic agents in interaction scenarios to further customize the system according to children’s needs. We perform extensive evaluations of the developed modules, achieving state-of-the-art results on both the children’s action BabyRobot dataset and the children’s emotion EmoReact dataset. Finally, we demonstrate the robustness and effectiveness of the IL system for action recognition by conducting a thorough experimental analysis for various conditions and parameters.

show abstract

“…Memory-based approaches, on the other hand, store previous experiences in memory and rehearse or indirectly use them in order to avoid forgetting them. To this end, the experiences themselves can be stored ( Robins, 1993 ) or a generative model can be trained to generate pseudo-experiences ( Robins, 1995 ) to rehearse experiences; or an episodic or semantic memory can be learned to retain information for longer terms and to interpret new experiences in the context of such a memory ( Hassabis et al, 2017 ; Churamani and Gunes, 2020 ). Finally, in model-extension approaches, the model (the network architecture) can be extended itself to accommodate the required capacity for the new task or experience ( Draelos et al, 2017 ).…”

Section: Related Workmentioning

confidence: 99%

“…Doğan et al (2018a , b) introduced solutions for addressing lifelong learning of context in robots continually encountering new situations through their lifetimes. As a last example, Churamani and Gunes (2020) proposed a memory-based solution for continual learning of facial expressions that can potentially be used by a humanoid robot to sense and continually learn its user’s affective states.…”

Section: Related Workmentioning

confidence: 99%

Mind Your Manners! A Dataset and a Continual Learning Approach for Assessing Social Appropriateness of Robot Actions

2022

Self Cite

View full text Add to dashboard Cite

To date, endowing robots with an ability to assess social appropriateness of their actions has not been possible. This has been mainly due to (i) the lack of relevant and labelled data and (ii) the lack of formulations of this as a lifelong learning problem. In this paper, we address these two issues. We first introduce the Socially Appropriate Domestic Robot Actions dataset (MANNERS-DB), which contains appropriateness labels of robot actions annotated by humans. Secondly, we train and evaluate a baseline Multi Layer Perceptron and a Bayesian Neural Network (BNN) that estimate social appropriateness of actions in MANNERS-DB. Finally, we formulate learning social appropriateness of actions as a continual learning problem using the uncertainty of Bayesian Neural Network parameters. The experimental results show that the social appropriateness of robot actions can be predicted with a satisfactory level of precision. To facilitate reproducibility and further progress in this area, MANNERS-DB, the trained models and the relevant code are made publicly available at https://github.com/jonastjoms/MANNERS-DB.

show abstract

CLIFER: Continual Learning with Imagination for Facial Expression Recognition

Cited by 25 publications

References 33 publications

Spatio-Temporal Analysis of Facial Actions using Lifecycle-Aware Capsule Networks

Spatio-Temporal Analysis of Facial Actions using Lifecycle-Aware Capsule Networks

Visual Robotic Perception System with Incremental Learning for Child–Robot Interaction Scenarios

Mind Your Manners! A Dataset and a Continual Learning Approach for Assessing Social Appropriateness of Robot Actions

Contact Info

Product

Resources

About