Infant AFAR: Automated facial action recognition in infants

Abstract

Automated detection of facial action units in infants is challenging. Infant faces have different proportions, less texture, fewer wrinkles and furrows, and unique facial actions relative to adults. For these and related reasons, action unit (AU) detectors that are trained on adult faces may generalize poorly to infant faces. To train and test AU detectors for infant faces, we trained convolutional neural networks (CNN) in adult video databases and fine-tuned these networks in two large, manually annotated, infant video databases that differ in context, head pose, illumination, video resolution, and infant age. AUs were those central to expression of positive and negative emotion. AU detectors trained in infants greatly outperformed ones trained previously in adults. Training AU detectors across infant databases afforded greater robustness to between-database differences than did training database specific AU detectors and outperformed previous state-of-the-art in infant AU detection. The resulting AU detection system, which we refer to as Infant AFAR (Automated Facial Action Recognition), is available to the research community for further testing and applications in infant emotion, social interaction, and related topics.

Automated measurement of infant and mother Duchenne facial expressions in the Face‐to‐Face/Still‐Face

“…The mean free‐margin kappa for infant AUs ( κ free = 0.85) were reported in Onal Ertuğrul et al. (2022). All free‐margin kappas were above 0.80 (see first two rows of Table 2).…”

“…Infants' and mothers' synchronized video‐recordings were submitted to the Automated Facial Affect Recognition (AFAR; Onal Ertuğrul, Cohn, et al., 2019, Onal Ertuğrul, Jeni, et al., 2019, 2022) computer vision software, an automated tool for detecting FACS AUs in the infant and mother faces. First, the Zface module (Jeni et al., 2017) of AFAR toolbox was used for automatic face tracking and registration.…”

“…Mothers' facial images were then input to the Adult AFAR network, which is a convolutional neural network that was trained on adult faces (Onal Ertuğrul, Jeni, et al., 2019). Infants' facial images were input to the Infant AFAR network that was trained on infant faces (Onal Ertuğrul et al., 2022). The AFAR network output the probability of the facial AUs (i.e., AU4, AU6, AU12, AU20) for each video frame.…”

Emergence of 3D body model-based infant health evaluation: a review

“…3). Fang et al 51 Onal Ertugrul et al 52 developed an infant action unit (AU) detector, InfantAFAR, using convolutional neural networks (Fig. 4).…”

An ensemble multi-stream classifier for infant needs detection

“…Also in [36] the authors studied how recognition of infants' emotion is necessary for the overall development, through audio / cry signals. The authors in [37] proposed Automated Facial Action Recognition (AFAR), trained using CNN across infant databases performed better compared to training database specific AU detectors which was trained on adult faces.…”