Recognition of facial expression of fetuses by artificial intelligence (AI)

Abstract

Objectives The development of the artificial intelligence (AI) classifier to recognize fetal facial expressions that are considered as being related to the brain development of fetuses as a retrospective, non-interventional pilot study. Methods Images of fetal faces with sonography obtained from outpatient pregnant women with a singleton fetus were enrolled in routine conventional practice from 19 to 38 weeks of gestation from January 1, 2020, to September 30, 2020, with completely de-identified data. The images were classified into seven categories, such as eye blinking, mouthing, face without any expression, scowling, smiling, tongue expulsion, and yawning. The category in which the number of fetuses was less than 10 was eliminated before preparation. Next, we created a deep learning AI classifier with the data. Statistical values such as accuracy for the test dataset and the AI confidence score profiles for each category per image for all data were obtained. Results The number of fetuses/images in the rated categories were 14/147, 23/302, 33/320, 8/55, and 10/72 for eye blinking, mouthing, face without any expression, scowling, and yawning, respectively. The accuracy of the AI fetal facial expression for the entire test data set was 0.985. The accuracy/sensitivity/specificity values were 0.996/0.993/1.000, 0.992/0.986/1.000, 0.985/1.000/0.979, 0.996/0.888/1.000, and 1.000/1.000/1.000 for the eye blinking, mouthing, face without any expression, scowling categories, and yawning, respectively. Conclusions The AI classifier has the potential to objectively classify fetal facial expressions. AI can advance fetal brain development research using ultrasound.

Toward deep observation: A systematic survey on artificial intelligence techniques to monitor fetus via ultrasound images

“…For the first time, 4D US images were utilized using the multi-classification method to identify fetus facial expression into Sad, Normal, and Happy as seen in ( Dave and Nadiad, 2015 ). Study ( Miyagi et al., 2021 ) used 2D US images in the second and third trimesters to classify fetal facial expression into eye blinking, mouthing without any expression, scowling, and yawning.…”

“…Of these studies (n = 10, 9.34%), we did not find any public dataset available online. However, within the private dataset, image augmentation was applied in (n = 3, 2.8%) studies ( Chen et al., 2020c ; Miyagi et al., 2021 ; Singh et al., 2021a ). The k-fold cross-validation was employed in ( Lei et al., 2014 ; Singh et al., 2021a ; Wang et al., 2021 ).…”

“…DL was used heavily in all fetal organ experiments including general fetus issues, fetus head, fetus face, fetus heart, and fetus abdomen. Furthermore, as seen in most of the models (n = 74, 69.1%), Convolutional Neural Network (CNN) used as a backbone of deep learning and is used in all tasks including classification, segmentation, and object detection ( Aji et al., 2019 ; Al-Bander et al., 2020 ; Andriani and Mardhiyah, 2019 ; Arnaout et al., 2021 ; Brahma et al., 2021 ; Budd et al., 2019 ; Cai et al., 2020 , 2018 ; Chen et al., 2014 , 2017 , 2020a , 2020b , 2020c ; Deepika et al., 2021 ; Desai et al., 2020 ; Dinesh Simon and Kavitha, 2021 ; Dong et al., 2020 ; Dozen et al., 2020 ; Droste et al., 2020 ; Fathimuthu Joharah and Mohideen, 2020 ; Fiorentino et al., 2021 ; Gao and Noble, 2019 ; Gong et al., 2020 ; Huang et al., 2018 ; Jang et al., 2017 ; Kim et al., 2018 , 2019a ; Komatsu et al., 2021b ; Li et al., 2020 ; Li et al., 2017 ; Lin et al., 2019a , 2019 ; Liu et al., 2020 ; Liu et al., 2019 ; Miyagi et al., 2021 ; Namburete and Noble, 2013 ; Nie et al., 2015a ; Patra et al., 2017 ; Perez-Gonzalez et al., 2020 ; Prieto et al., 2021 ; Pu et al., 2021 ; Qiao and Zulkernine, 2020 ; Qu et al., 2020a ; Ravishankar et al., 2016 ; Ryou et al., 2016 , 2019 ; Sahli et al., 2020 ; Singh et al., 2021a , 2021b ; Sobhaninia et al., 2019 , …”

Artificial intelligence to understand fluctuation of fetal brain activity by recognizing facial expressions

“…This was performed in MP4 format at a rate of 10.008 frames per second using a GE Voluson E10 BT20 (GE Healthcare). Informed consent was obtained from all participants at Miyake Clinic between February 1 and December 31, 2021, with completely deidentified data enrolled 14,15 . We transferred the videos to an offline AI system at Medical Data Labo, Japan.…”

Artificial intelligence in obstetrics

“…In particular, the application of ultrasound artificial intelligence may be useful in probe guidance, fetal biometrics plane finder, anomaly scan completeness, anomaly highlighting, and cyst classification [ 63 ]. One example is the effort to detect changes in fetal heart structures or facial expressions using artificial intelligence technologies [ 64 , 65 ].…”