Development and External Validation of a Deep Learning Algorithm to Identify and Localize Subarachnoid Hemorrhage on CT Scans

Abstract

Objective:In medical imaging, a limited number of trained deep learning algorithms have been externally validated and released publicly. We hypothesized that a deep learning algorithm can be trained to identify and localize subarachnoid haemorrhage (SAH) on head computed tomography (CT) scans, and that the trained model performs satisfactorily when tested using external and real-world data.Methods:We used non-contrast head CT images of patients admitted Helsinki University Hospital between 2012 and 2017. We manually segmented (i.e. delineated) SAH on 90 head CT scans, and used the segmented CT scans together with 22 negative (no SAH) control CT scans in training an open-source convolutional neural network (U-Net) to identify and localize SAH. We then tested the performance of the trained algorithm by using external datasets (137 SAH and 1242 control cases) collected in two foreign countries, and also by creating a dataset of consecutive emergency head CT scans (8 SAH and 511 control cases) performed during on call hours in 5 different domestic hospitals in September 2021. We assessed the algorithm’s capability to identify SAH by calculating patient- and slice-level performance metrics, such as sensitivity and specificity.Results:In the external validation set of 1379 cases, the algorithm identified 136 out of 137 SAH cases correctly (sensitivity 99.3%, specificity 63.2%). Of the 49064 axial head CT slices, the algorithm identified and localized SAH in 1845 out of 2110 slices with SAH (sensitivity 87.4%, specificity 95.3%). Of 519 consecutive emergency head CT scans imaged in September 2021, the algorithm identified all 8 SAH cases correctly (sensitivity 100.0%, specificity 75.3%). The slice-level (27167 axial slices in total) sensitivity and specificity were 87.3% and 98.8%, as the algorithm identified and localized SAH in 58 out of 77 slices with SAH. The performance of the algorithm can be tested on through a webservice.Conclusions:We show that the shared algorithm identifies SAH cases with a high sensitivity, and that the slice-level specificity is high. In addition to openly sharing a high-performing deep learning algorithm, our work presents infrequently used approaches in designing, training, testing and reporting deep learning algorithms developed for medical imaging diagnostics.Classification of Evidence:This study provides Class III evidence a deep learning algorithm correctly identifies the presence of subarachnoid hemorrhage on CT scan.

Deep Learning Algorithms for Brain Imaging

“…Thanellas et al 6 have made the algorithm available for download and have provided an online tool to upload fully anonymized head CT scans. The availability of these tools provides a first-hand opportunity to assess and familiarize researchers with deep learning methods.…”

“…Another critical aspect of paving the way to translate AI for clinical use is quantifying uncertainty, which can enhance clinical interpretation and indicate whether further investigation of the results by a clinician is warranted. 1 Thanellas et al 6 have made the algorithm available for download and have provided an online tool to upload fully anonymized head CT scans. The availability of these tools provides a first-hand opportunity to assess and familiarize researchers with deep learning methods.…”

“…Typically, a large number of images are required for training an algorithm in order for it to distinguish between images with small but potentially significant differences in features. 8 Compared with previous studies, instead of training the algorithm using a large number of images, Thanellas et al 6 trained it with pixel-level data instead of slice-level-annotated images to adapt their study for a smaller sample size. 9 As a result, the algorithm classifies every single pixel in axial images presented to it and predicts the presence of SAH at a patient level and slice level.…”

“…A study published in this issue of Neurology ® by Thanellas et al 6 provides a detailed investigation of how a deep learning algorithm was trained and applied in a patient population to identify and localize subarachnoid hemorrhage (SAH) based on head CT scans. SAH represents intracranial bleeding which, if untreated, has a 75% mortality rate within 1 year.…”

“…Similarly, it is important to understand the patient population in which a particular algorithm is developed. Thanellas et al 6 trained their algorithm exclusively on a Finnish population. Underlying data are likely influenced by genetic, geographic, social, and environmental factors.…”