Ongoing Face Recognition Vendor Test (FRVT) part 6B :

Abstract

This is the second of a series of reports on the performance of face recognition algorithms on faces occluded by protective face masks [2] commonly worn to reduce inhalation and exhalation of viruses. Inspired by the COVID-19 pandemic response, this is a continuous study being run under the Ongoing Face Recognition Vendor Test (FRVT) executed by the National Institute of Standards and Technology (NIST). In our first report [8], we tested "pre-pandemic" algorithms that were already submitted to FRVT 1:1 prior to mid-March 2020. This report augments its predecessor with results for more recent algorithms provided to NIST after mid-March 2020. While we do not have information on whether or not a particular algorithm was designed with face coverings in mind, the results show evidence that a number of developers have adapted their algorithms to support face recognition on subjects potentially wearing face masks. The algorithms tested were one-to-one algorithms submitted to the FRVT 1:1 Verification track. Future editions of this document will also report accuracy of one-to-many algorithms. WHAT'S NEW This report includes Results from evaluating 65 face recognition algorithms provided to NIST since mid-March 2020 Assessment of when both the enrollment and verification images are masked (in addition to when only the verification image is masked) Results for red and white colored masks (in addition to light-blue and black) Cumulative results for 152 algorithms evaluated to date (submitted both prior to and after mid-March 2020) MOTIVATION Traditionally, face recognition systems (in cooperative settings) are presented with mostly nonoccluded faces, which include primary facial features such as the eyes, nose, and mouth. However, there are a number of circumstances in which faces are occluded by masks such as in pandemics, medical settings, excessive pollution, or laboratories. Inspired by the COVID-19 pandemic response, the widespread requirement that people wear protective face masks in public places has driven a need to understand how cooperative face recognition technology deals with occluded faces, often with just the periocular area and above visible. An increasing number of research publications have surfaced on the topic of face recognition on people wearing masks along with face-masked research datasets [10]. A number of commercial providers have announced the availability of face recognition algorithms capable of handling face masks, and this report documents performance results for 65 algorithms submitted to NIST after mid-March 2020. This report includes results for all algorithms evaluated to date. At the time of this writing, we are not aware of any large-scale, independent, and publicly reported evaluation on the effects of face mask occlusion on face recognition. WHAT WE DID The NIST Information Technology Laboratory (ITL) quantified the accuracy of face recognition algorithms on faces occluded by masks applied digitally to a large set of photos that has been used in an FRVT verification benchmark since 201...

Extended evaluation of the effect of real and simulated masks on face recognition performance

“…We use the synthetic mask generation method described by the NIST report [14]. The synthetic generation method depends on the Dlib toolkit [31] to detect and extract 68 facial landmarks from a face image.…”

Masked Face Recognition Using Deep Learning: A Review

“…The NIST [3] recently reviewed the performance of FR algorithms before and after the COVID-19 pandemic. They evaluated the existing algorithms (pre-pandemic) after tuning them to deal with masked or concluded faces.…”

“…Recently, the National Institute for Standards and Technology (NIST) [3] presented the performance of a set of face recognition algorithms developed and tuned after the COVID-19 pandemic (post-COVID- 19), which follows their first study on pre-COVID-19 algorithms [4]. They concluded that the majority of recognition algorithms evaluated after the pandemic still show a performance degradation when faces are masked.…”

Transdisciplinary AI Observatory—Retrospective Analyses and Future-Oriented Contradistinctions

“…Finally, concerning AI risk Id, we observe one main failure cluster which is connected to unanticipated post-deployment usage modes and contexts which also includes eventual complications within unusual interactions of the AI system in a dynamically changing environment. Notable examples are failures of facial recognition AI linked to COVID-19 causing the widespread use of facial masks [106][107][108], the invariant responses of natural language processing systems when faced with nonsensical instead of usual meaningful queries [109] (disclosing the low level of understanding) and the AI-based censorship of a picture displaying ancient slavery settings due to a forerunning misclassification labelling the sample as displaying nudity [110]. Other cases include unknown latent biases in medical AI [111] and other forms of biases in medical AI that unfold post-deployment as a function of geographical factors [112].…”