Face Recognition and Tracking Framework for Human–Robot Interaction

Abstract

Recently, face recognition became a key element in social cognition which is used in various applications including human–robot interaction (HRI), pedestrian identification, and surveillance systems. Deep convolutional neural networks (CNNs) have achieved notable progress in recognizing faces. However, achieving accurate and real-time face recognition is still a challenging problem, especially in unconstrained environments due to occlusion, lighting conditions, and the diversity in head poses. In this paper, we present a robust face recognition and tracking framework in unconstrained settings. We developed our framework based on lightweight CNNs for all face recognition stages, including face detection, alignment and feature extraction, to achieve higher accuracies in these challenging circumstances while maintaining the real-time capabilities required for HRI systems. To maintain the accuracy, a single-shot multi-level face localization in the wild (RetinaFace) is utilized for face detection, and additive angular margin loss (ArcFace) is employed for recognition. For further enhancement, we introduce a face tracking algorithm that combines the information from tracked faces with the recognized identity to use in the further frames. This tracking algorithm improves the overall processing time and accuracy. The proposed system performance is tested in real-time experiments applied in an HRI study. Our proposed framework achieves real-time capabilities with an average of 99%, 95%, and 97% precision, recall, and F-score respectively. In addition, we implemented our system as a modular ROS package that makes it straightforward for integration in different real-world HRI systems.

Face Identification Based on Active Facial Patches Using Multi-Task Cascaded Convolutional Networks

“…The illustrated approach in [31] aims to resolve these issues by evaluating Neo-natal Intensive Care Unit (NICU) face detection technology settings [32]. NICU face models demonstrate fine-tuning can enhance the performance of the robustness of new face detectors in complex NICU environments [33].…”

Recent advancements in multimodal human–robot interaction

“… Multimodal HRI (Castillo et al, 2013 ; Yu et al, 2016 ; Lannoy et al, 2017 ; Legrand et al, 2018 ; Stephens-Fripp et al, 2018 ; Mohebbi, 2020 ; Khalifa et al, 2022 ; Otálora et al, 2022 ; Strazdas et al, 2022 ). …”

“…Khalifa et al ( 2022 ) present a robust framework for face tracking and identification in unrestricted environments, designing their framework based on lightweight CNNs to increase accuracy while preserving real-time capabilities essential for HRI systems. Shenoy et al ( 2021 ) improve the interaction capabilities of Nao humanoid robots by combining detection models for facial expression and speech quality, using the microphone and camera to assess pain and mood in children receiving procedural therapy.…”

Special Issue on Trends and Challenges in Robotic Applications

“…Computer vision focuses on enabling computers to interpret and understand visual information from the world, including objects and humans [15]. It involves the development of algorithms and techniques that allow machines to process, analyze and extract meaningful insights from images and videos.…”