Effectiveness evaluation of computer-aided diagnosis system for the diagnosis of thyroid nodules on ultrasound

Abstract

Background: More and more automated efficient ultrasound image analysis techniques, such as ultrasound-based computer-aided diagnosis system (CAD), were developed to obtain accurate, reproducible, and more objective diagnosis results for thyroid nodules. So far, whether the diagnostic performance of existing CAD systems can reach the diagnostic level of experienced radiologists is still controversial. The aim of the meta-analysis was to evaluate the accuracy of CAD for thyroid nodules’ diagnosis by reviewing current literatures and summarizing the research status. Methods: A detailed literature search on PubMed, Embase, and Cochrane Libraries for articles published until December 2018 was carried out. The diagnostic performances of CAD systems vs radiologist were evaluated by meta-analysis. We determined the sensitivity and the specificity across studies, calculated positive and negative likelihood ratios and constructed summary receiver-operating characteristic (SROC) curves. Meta-analysis of studies was performed using a mixed-effect, hierarchical logistic regression model. Results: Five studies with 536 patients and 723 thyroid nodules were included in this meta-analysis. The pooled sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, and diagnostic odds ratio (DOR) for CAD system were 0.87 (95% confidence interval [CI], 0.73–0.94), 0.79 (95% CI 0.63–0.89), 4.1 (95% CI 2.5–6.9), 0.17 (95% CI 0.09–0.32), and 25 (95% CI 15–42), respectively. The SROC curve indicated that the area under the curve was 0.90 (95% CI 0.87–0.92). The pooled sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, and DOR for experienced radiologists were 0.82 (95% CI 0.69–0.91), 0.83 (95% CI 0.76–0.89), 4.9 (95% CI 3.4–7.0), 0.22 (95% CI 0.12–0.38), and 23 (95% CI 11–46), respectively. The SROC curve indicated that the area under the curve was 0.96 (95% CI 0.94–0.97). Conclusion: The sensitivity of the CAD system in the diagnosis of thyroid nodules was similar to that of experienced radiologists. However, the CAD system had lower specificity and DOR than experienced radiologists. The CAD system may play the potential role as a decision-making assistant alongside radiologists in the thyroid nodules’ diagnosis. Future technical improvements would be helpful to increase the accuracy as well as diagnostic efficiency.

S-Detect Software vs. EU-TIRADS Classification: A Dual-Center Validation of Diagnostic Performance in Differentiation of Thyroid Nodules

“…Differentiation between benign and malignant TNs is challenging, even for experienced specialists. Accordingly, some recent studies have focused on S-Detect system efficacy in comparison to evaluations performed by experienced medical physicians or other types of evaluations using CAD [14,15]. A recent meta-analysis performed by Zhao et al including 723 lesions from five studies revealed that the sensitivity of the CAD was comparable to the assessment provided by experienced radiologists (0.87 vs. 0.88) but presented lower specificity (0.79 vs. 0.92).…”

“…A recent meta-analysis performed by Zhao et al including 723 lesions from five studies revealed that the sensitivity of the CAD was comparable to the assessment provided by experienced radiologists (0.87 vs. 0.88) but presented lower specificity (0.79 vs. 0.92). However, there is a need for larger sample-size prospective studies to verify the clinical usefulness of CAD in a clinical setting, together with further technical improvements of the software [15].…”

“…To overcome this difficulty, computer-aided diagnosis (CAD) systems are gaining interest for US image analysis and are developed on the basis of statistical data mining algorithms, collected from medical center databases. It enables non-invasive judgement on benignity or malignancy of TNs on the basis of US image analysis [14,15]. The purpose of CAD is to increase the diagnostic confidence, achieve interpretation constancy of US features, and eliminate the inter-observer variability in order to increase diagnostic accuracy, especially when the examination is performed by ultrasonographers outside referenced centers for thyroid cancer diagnostics.…”

“…Thus, it is very useful for discarding the benign cases and reducing the number of biopsies [17]. CAD is supposed to be an applicable tool in TN diagnostics and clinical decision-making for medical practitioners with basic US skills [15,16]. It may be useful especially for less experienced operators, increasing the specificity of the examination [18].…”

The Diagnostic Efficiency of Ultrasound Computer–Aided Diagnosis in Differentiating Thyroid Nodules: A Systematic Review and Narrative Synthesis

“…Existing systematic reviews have been based on analyzing different textural techniques and machine learning algorithms and classifiers more on a biomedical engineering perspective rather than clinical applicability [13,17,18,19]. A recent systematic review and meta-analysis study focused on the diagnostic performance of mainly the Samsung S-Detect CAD software in comparison with that of radiologists for the differentiation of thyroid nodules in clinical settings [20]. As various CAD systems for ultrasound feature assessment of thyroid nodules have been developed by various researchers and clinicians in different parts of the world, there are multiple factors that can influence their diagnostic performance.…”