Multi-Scale Part-Based Syndrome Classification of 3D Facial Images

Abstract

Identification and delineation of craniofacial characteristics support the clinical and molecular diagnosis of genetic syndromes. Deep learning (DL) frameworks for syndrome identification from 2D facial images are trained on large clinical datasets using standard convolutional neural networks for classification. In contrast, despite the increased availability of 3D scanners in clinical setups, similar frameworks remain absent for 3D facial photographs. The main challenges involve working with smaller datasets and the need for DL operations applicable to 3D geometric data. Therefore, to date, most 3D methods refrain from working across multiple syndromic groups and/or are solely based on traditional machine learning. The first contribution of this work is the use of geometric deep learning with spiral convolutions in a triplet-loss architecture. This geometric encoding (GE) learns a lower dimensional metric space from 3D facial data that is used as input to linear discriminant analysis (LDA) performing multiclass classification. Benchmarking is done against principal component analysis (PCA), a common technique in 3D facial shape analysis, and related work based on 65 distinct 3D facial landmarks as input to LDA. The second contribution of this work involves a part-based implementation to 3D facial shape analysis and multi-class syndrome classification, and this is applied to both GE and PCA. Based on 1,786 3D facial photographs of controls and individuals from 13 different syndrome classes, a five-fold cross-validation was used to investigate both contributions. Results indicate that GE performs better than PCA as input to LDA, and this especially so for more compact (lower dimensional) spaces. In addition, a part-based approach increases performance significantly for both GE and PCA, with a more significant improvement for the latter. I.e., this contribution enhances the power of the dataset. Finally, and interestingly, according to ablation studies within the part-based approach, the upper lip is the most distinguishing facial segment for classifying genetic syndromes in our dataset, which follows clinical expectation. This work stimulates an enhanced use of advanced part-based geometric deep learning methods for 3D facial imaging in clinical genetics.

Computer aided diagnosis of neurodevelopmental disorders and genetic syndromes based on facial images – A systematic literature review

“…Whereas the works based on 3D images, mentioned so far, used only traditional machine learning techniques, Mahdi et al [75] used geometric deep learning (GDL) to extract features directly from the non-Euclidean facial surfaces for classification of 13 different syndromes. In addition, part-based and full-face approaches were tested.…”

“…Traditional CNN architectures are designed for 2D input images. Instead, [75] used spiral convolutional operators on a non-Euclidean domain. Although [70] also used deep learning and 3D reconstruction, the deep features were extracted from the 2D original images.…”

A 3D Clinical Face Phenotype Space of Genetic Syndromes using a Triplet-Based Singular Geometric Autoencoder

“…Our proposed model consists of three main components. The first is a triplet-based encoder which was used in the recent syndrome classification work in [25] to optimize the distances among individuals belonging to different syndrome groups. In the triplet-loss function, the focus is on learning the CFPS such that the distances are a measure of similarity and group membership and therefore it contributes to the classification and clustering power of the space.…”

“…6) Classification: In this study, we performed a one vs all classification of syndromes, where a specific syndrome is preselected, and patients are classified as either having or not having that condition, constituting a binary classification or a syndrome identification task [4], [25], [26]. We also conducted a syndrome identification task, answering the question: Given a patient, which syndrome class is most likely?…”

“…Bannister et al [24] classified 47 genetic syndromes from facial surface scans using a normalizing flow architecture. Mahdi et al [25] introduced a multi-scale part-based metric learner for classification of 14 syndromes form surface scans represented by ∼8000 landmarks. The concept of CFPS was first introduced in [2] where principal component (PC) scores, representing variation in landmark coordinates and local pixel intensity variation around the landmarks, were derived from 2D images of 8 disorders and subjected to the linear largest margin nearest neighbor metric learner.…”