Detecting Essential Landmarks Directly in Thermal Images for Remote Body Temperature and Respiratory Rate Measurement With a Two-Phase System

Abstract

Infrared thermographs (IRTs, also called thermal cameras) have been used to remotely measure elevated body temperature (BT) and respiratory rate (RR) during infectious disease outbreaks, such as COVID-19. To facilitate the fast measurement of BT and RR using IRTs in densely populated venues, it is desirable to have IRT algorithms that can automatically identify the best facial locations in thermal images to extract these vital signs. The IEC 80601-2-59:2017 standard suggests that the regions medially adjacent to the inner canthi of the eyes are robust BT measurement sites. The nostril regions, on the other hand, are often used for RR estimation. However, it is more difficult to automatically identify inner canthi and nostrils in thermal images than in visible-light images, which are rich with exploitable features. In this paper, a unique system that can detect inner canthi and outer nostril edges directly in thermal images in two phases is introduced. In Phase I, original thermal images were processed in four different ways to enhance facial features to facilitate inner canthus and nostril detection. In Phase II, landmarks of the inner canthi and outer nostril edges were detected in two steps: (1) face detection using the Single Shot Multibox Detector (SSD) and (2) facial landmark detection to locate the inner canthi and outer nostril edges. The face detection, facial landmark detection, and overall system accuracies were evaluated using the intersection over union, normalized Euclidean distance, and success detection rate metrics on a set of 36 thermal images collected from 12 subjects using three different IRTs. Additional validation was performed on a subset of 40 random thermal images from the publicly available Tufts Face Database. The results revealed that the processed images-referred to as ICLIP images-yielded the highest landmark localization accuracy from the four types of processed thermal images, verifying that the system can automatically and accurately estimate the inner canthus and nostril locations in thermal images. The proposed system can be applied in IRT algorithms to provide reliable temperature measurements and RR estimates during infectious disease outbreaks.

Eye Recognition by YOLO for Inner Canthus Temperature Detection in the Elderly Using a Transfer Learning Approach

“…Besides the size of our used dataset, it also contains different face poses, enabling the model to identify the eye region in straight-facing images and in rotated-facing images ( Figure 18 ). This is considered a limitation in the system developed by Lazri et al [ 18 ]. In the case of the rotated photos, the detection of the temperature of one inner canthus is sufficient to measure the person’s temperature.…”

“…Additionally, our work could be implemented in a real-time temperature monitoring system, since we were able to detect the eyes in a video scene with a high speed, reaching 150 FPS in the trained YOLOv7 model, as shown in Figure 19 . This is compared to a maximum of 146 FPS in [ 18 ] when testing a pre-trained single shot multi-box detector (SSD). Regarding accuracy, a comparison between our model and that of [ 18 ] is inapplicable, since there, the SSD model is pre-trained on visible images and IoU was as an accuracy metric, whereas our model is trained solely on thermal images, with %mAP@0.5 taken as an accuracy measure.…”

“…This is compared to a maximum of 146 FPS in [ 18 ] when testing a pre-trained single shot multi-box detector (SSD). Regarding accuracy, a comparison between our model and that of [ 18 ] is inapplicable, since there, the SSD model is pre-trained on visible images and IoU was as an accuracy metric, whereas our model is trained solely on thermal images, with %mAP@0.5 taken as an accuracy measure.…”

“…Their work was performed by training the OpenPose detector on visible and not thermal images, making it not reliable enough [ 17 ]. Finally, the most recent attempt was that of Lazri et al [ 18 ], who worked on detecting inner canthi and nostrils to measure body temperature and respiration rate. They used the pre-trained single shot multibox detector (SSD) to detect faces in thermal images, then they searched for the landmarks using Kazemi and Sullivan’s publicly available algorithm.…”

Identity-Privacy Protection for Facial-rPPG Based Smart Health Research

“…For example, the study in [13] captured videos of hands to extract physiological signals. Infrared imaging was used in [14] to extract respiratory rate. It is more difficult to directly recognize the subjects' identities from hand videos or infrared videos than from facial videos.…”