Development and external validation of automated detection, classification, and localization of ankle fractures: inside the black box of a convolutional neural network (CNN)

Abstract

Purpose Convolutional neural networks (CNNs) are increasingly being developed for automated fracture detection in orthopaedic trauma surgery. Studies to date, however, are limited to providing classification based on the entire image—and only produce heatmaps for approximate fracture localization instead of delineating exact fracture morphology. Therefore, we aimed to answer (1) what is the performance of a CNN that detects, classifies, localizes, and segments an ankle fracture, and (2) would this be externally valid? Methods The training set included 326 isolated fibula fractures and 423 non-fracture radiographs. The Detectron2 implementation of the Mask R-CNN was trained with labelled and annotated radiographs. The internal validation (or ‘test set’) and external validation sets consisted of 300 and 334 radiographs, respectively. Consensus agreement between three experienced fellowship-trained trauma surgeons was defined as the ground truth label. Diagnostic accuracy and area under the receiver operator characteristic curve (AUC) were used to assess classification performance. The Intersection over Union (IoU) was used to quantify accuracy of the segmentation predictions by the CNN, where a value of 0.5 is generally considered an adequate segmentation. Results The final CNN was able to classify fibula fractures according to four classes (Danis-Weber A, B, C and No Fracture) with AUC values ranging from 0.93 to 0.99. Diagnostic accuracy was 89% on the test set with average sensitivity of 89% and specificity of 96%. External validity was 89–90% accurate on a set of radiographs from a different hospital. Accuracies/AUCs observed were 100/0.99 for the ‘No Fracture’ class, 92/0.99 for ‘Weber B’, 88/0.93 for ‘Weber C’, and 76/0.97 for ‘Weber A’. For the fracture bounding box prediction by the CNN, a mean IoU of 0.65 (SD ± 0.16) was observed. The fracture segmentation predictions by the CNN resulted in a mean IoU of 0.47 (SD ± 0.17). Conclusions This study presents a look into the ‘black box’ of CNNs and represents the first automated delineation (segmentation) of fracture lines on (ankle) radiographs. The AUC values presented in this paper indicate good discriminatory capability of the CNN and substantiate further study of CNNs in detecting and classifying ankle fractures. Level of evidence II, Diagnostic imaging study.

Advancements in Artificial Intelligence for Foot and Ankle Surgery: A Systematic Review

“…Kitamura et al 26 internally validated 5 separate CNNs for detecting ankle fractures from plain radiographs and achieved a fair fracture detection accuracy of 81%. Prijs et al 44 internally and externally validated a DL model for detecting, classifying, and localizing ankle fractures from plain radiographs and achieved an excellent AUC of 0.92 and accuracy of 99% (classifying “no fracture”) on external validation. Guermazi et al 14 internally validated a DL model for detecting fractures from foot and ankle plain radiographs, which performed excellently with an AUC of 0.97, sensitivity per patient of 93%, and specificity per patient of 93%.…”

Künstliche Intelligenz in der Fuß- und Sprunggelenkchirurgie: aktuelle Konzepte

“…Prijs et al. internally and externally validated a DL model for detecting, classifying, and localizing ankle fractures from plain radiographs and achieved an excellent AUC of 0.92 and an accuracy of 99 % on external validation [ 13 ]. Guermazi et al.…”

CNN Attention Guidance for Improved Orthopedics Radiographic Fracture Classification

“…The second investigated dataset [24] is an ankle fracture dataset also collected at the FMC. The dataset contains a total…”

“…Our proposed attention guidance technique is formed as a network regularization mechanism comprising two parts: 1) an elaborate modification of the widely used state-of-the-art CNN architecture (e.g., using ResNet [23] as the backbone model) to accept human guided attention, and 2) a novel attention regularization loss function to incorporate the accepted guidance signals during network training. Extensive experiments on our proprietary scaphoid [4] and ankle fracture [24] classification datasets demonstrate that the proposed attention guidance method can lead to more accurately grounded fracture localization and better classification performance at the same time. The main contributions of this work are summarized as follows.…”