Automatic Recognition of Macaque Facial Expressions for Detection of Affective States

Abstract

Internal affective states produce external manifestations such as facial expressions. In humans, the Facial Action Coding System (FACS) is widely used to objectively quantify the elemental facial action-units (AUs) that build complex facial expressions. A similar system has been developed for macaque monkeys -the Macaque Facial Action Coding System (MaqFACS); yet unlike the human counterpart, which is already partially replaced by automatic algorithms, this system still requires labor-intensive coding. Here, we developed and implemented the first prototype for automatic MaqFACS coding. We applied the approach to the analysis of behavioral and neural data recorded from freely interacting macaque monkeys. The method achieved high performance in recognition of six dominant AUs, generalizing between conspecific individuals (Macaca mulatta) and even between species (Macaca fascicularis). The study lays the foundation for fully automated detection of facial expressions in animals, which is crucial for investigating the neural substrates of social and affective states. Significance StatementMaqFACS is a comprehensive coding system designed to objectively classify facial expressions based on elemental facial movements designated as Actions Units (AUs). It allows the comparison of facial expressions across individuals of same or different species based on manual scoring of videos, a laborand time-consuming process. We implemented the first automatic prototype for AUs coding in macaques. Using machine learning, we trained the algorithm on video-frames with AU labels, and showed that after parameter tuning, it classified six AUs in new individuals. Our method demonstrates concurrent validity with manual MaqFACS coding and supports the usage of automated MaqFACS. Such automatic coding is useful not only for social-and affective-neuroscience research but also for monitoring animal health and welfare.The archive "autoMaqFACS_code.zip" contains Matlab code for autoMaqFACS classification.

A novel method for automatically analysing the activity of fast-moving animals: a case study of Callimico goeldii monkeys housed in a zoological garden

“…In the literature, methods based on animal detection from video frames are most often used for the individual assessment of the behaviour of animals in cages, e.g. hens 32 , red foxes 33 , pigs in a pen 28 , 34 , 35 , monkeys 36 , 37 , and drosophila 38 . However, these methods introduce initial conditions, e.g.…”

“…However, these methods introduce initial conditions, e.g. they require the following: the presence of only one object in the cage 32 ; a model, a sample of training data, and a method training process 34 , 37 ; the need to have the camera pointed perpendicular to the floor to capture the entire pen 28 , 35 .…”

CalliFACS: The common marmoset Facial Action Coding System

“…long time and asynchronous video analysis, may challenge the potential of CalliFACS as a welfare tool, as many settings may require an immediate evaluation of facial indicators of welfare. As such, one of the next research steps after developing CalliFACS will be to automate the detection of AUs (recently published in rhesus and long-tailed macaques: [ 91 ], and in horses: [preprint: 92 ]). The automation of AnimalFACS will not only decrease the amount of work, time and training needed for use, but also allow the development of software applications where AUs can easily and quickly be identified and their meaning potentially extracted.…”

Probing the processing of facial expressions in monkeys via time perception and eye tracking

“…For human facial expressions, typical Fear, Happy, and Neutral facial expressions corresponding to the three categories were chosen as visual stimuli based on previous studies ( Angrilli et al, 1997 ; Calvo et al, 2013 ; Calvo & Nummenmaa, 2008 ; Gil & Droit-Volet, 2011 ; Yin et al, 2021 ). For monkey facial expressions, the three categories were chosen according to previous studies ( Liu et al, 2015 ; Maréchal et al, 2017 ; Micheletta et al, 2015 ; Morozov et al, 2021 ). Specifically, the Negative category included facial expressions with raised eyebrows and open mouth showing teeth (Threat, typically classified as “aggressive”); the Positive category included facial expressions with lip smacking (Lipsmack, classified as “friendly”); and the Neutral category included facial expressions with an overall relaxed expression (Neutral).…”