A novel end-to-end dual-camera system for eye gaze synchrony assessment in face-to-face interaction

Abstract

Quantification of face-to-face interaction can provide highly relevant information in cognitive and psychological science research. Current commercial glint-dependent solutions suffer from several disadvantages and limitations when applied in face-to-face interaction, including data loss, parallax errors, the inconvenience and distracting effect of wearables, and/or the need for several cameras to capture each person. Here we present a novel eye-tracking solution, consisting of a dual-camera system used in conjunction with an individually optimized deep learning approach that aims to overcome some of these limitations. Our data show that this system can accurately classify gaze location within different areas of the face of two interlocutors, and capture subtle differences in interpersonal gaze synchrony between two individuals during a (semi-)naturalistic face-to-face interaction.

Influence of autistic traits and communication role on eye contact behavior during face-to-face interaction

“…We have recently shown that an adaptation of current deep-learning methodologies for eye tracking 48 , 49 , together with a dual-camera system, can determine gaze to discriminate specific facial areas and quantify movement synchronization 42 . The camera system has been developed with neurodevelopmental disorders in mind; it is minimal, does not include any wearables, such as eye-tracking glasses, and is located on a table between two individuals without covering the face of the person in front of them, see Fig.…”

“…This error was lower error than in the previous implementation of the camera system, indicating the value of individual deep-learning calibration to actual face-to-face gaze. More detailed information about the eye-tracking technique can be found in Thorsson et al 42 . Missing data were defined as gaze that could not be estimated, and represented about 10.3% of the data collected in the experiment.…”

“…Gaze percentages were estimated as the average time spent looking at the eye area divided by the total time over the two trials when the participant described or guessed words. In order to estimate where the participants looked, facial planes (3D orientation and location) were first estimated based on facial landmark detection (see Thorsson et al 42 ; Appendix A: 2.1 Estimation of face plane). Gaze vectors were then estimated based on a 3D gaze model, that included the head pose and the eye and pupil location.…”

“…The gaze to the other person's face was estimated based on the average intersection of the two separate eye gaze vectors (from left and right eye) in the facial plane (see Supplementary Information and Thorsson et al 42 ; Appendix A: 2.2 Line-plane intersection). The classification of eye gaze was based on the distance between the average intersection of the other individual's facial plane (meaning that the area moved with the participants/experimenter) and the eyes and mouth landmarks (see Thorsson et al 42 and Supplementary Information ). Following the previous implementation with the camera system 42 we used a distance of 6.3 cm (~ 3°) to classify gaze to the eye area, see Fig.…”

“…We have designed, made, and developed a dual-camera system that accurately classifies gaze between interlocutors in face-to-face interactions 42 . The setup—called i + i—was developed with neurodevelopmental populations in mind, without additional cameras near the participants or wearables.…”