Using Bayesian neural networks with ARD input selection to detect malignant ovarian masses prior to surgery

Valentin

et al. 2010

BMC Med Res Methodol

Self Cite

BackgroundHitherto, risk prediction models for preoperative ultrasound-based diagnosis of ovarian tumors were dichotomous (benign versus malignant). We develop and validate polytomous models (models that predict more than two events) to diagnose ovarian tumors as benign, borderline, primary invasive or metastatic invasive. The main focus is on how different types of models perform and compare.MethodsA multi-center dataset containing 1066 women was used for model development and internal validation, whilst another multi-center dataset of 1938 women was used for temporal and external validation. Models were based on standard logistic regression and on penalized kernel-based algorithms (least squares support vector machines and kernel logistic regression). We used true polytomous models as well as combinations of dichotomous models based on the 'pairwise coupling' technique to produce polytomous risk estimates. Careful variable selection was performed, based largely on cross-validated c-index estimates. Model performance was assessed with the dichotomous c-index (i.e. the area under the ROC curve) and a polytomous extension, and with calibration graphs.ResultsFor all models, between 9 and 11 predictors were selected. Internal validation was successful with polytomous c-indexes between 0.64 and 0.69. For the best model dichotomous c-indexes were between 0.73 (primary invasive vs metastatic) and 0.96 (borderline vs metastatic). On temporal and external validation, overall discrimination performance was good with polytomous c-indexes between 0.57 and 0.64. However, discrimination between primary and metastatic invasive tumors decreased to near random levels. Standard logistic regression performed well in comparison with advanced algorithms, and combining dichotomous models performed well in comparison with true polytomous models. The best model was a combination of dichotomous logistic regression models. This model is available online.ConclusionsWe have developed models that successfully discriminate between benign, borderline, and invasive ovarian tumors. Methodologically, the combination of dichotomous models was an interesting approach to tackle the polytomous problem. Standard logistic regression models were not outperformed by regularized kernel-based alternatives, a finding to which the careful variable selection procedure will have contributed. The random discrimination between primary and metastatic invasive tumors on temporal/external validation demonstrated once more the necessity of validation studies.

Section: Methodsmentioning

confidence: 99%

Polytomous diagnosis of ovarian tumors as benign, borderline, primary invasive or metastatic: development and validation of standard and kernel-based risk prediction models

Valentin

et al. 2010

BMC Med Res Methodol

Self Cite

“…Four different types of mathematical models were developed to evaluate which type of model did best (scoring system, logistic regression models, artificial neural networks (ANN), and vector machine models). On internal and temporal validation in the centers that previously developed the models, they had excellent diagnostic performance (16,(22)(23)(24)(25). After temporal validation, all models had similar performance (AUCs between 0.945 and 0.950).…”

Section: Introductionmentioning

confidence: 95%

External Validation of Diagnostic Models to Estimate the Risk of Malignancy in Adnexal Masses

Clinical Cancer Research

Bourne

et al. 2012

Self Cite

Purpose: To externally validate and compare the performance of previously published diagnostic models developed to predict malignancy in adnexal masses.Experimental Design: We externally validated the diagnostic performance of 11 models developed by the International Ovarian Tumor Analysis (IOTA) group and 12 other (non-IOTA) models on 997 prospectively collected patients. The non-IOTA models included the original risk of malignancy index (RMI), three modified versions of the RMI, six logistic regression models, and two artificial neural networks. The ability of the models to discriminate between benign and malignant adnexal masses was expressed as the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and likelihood ratios (LR þ , LR À ). Results: Seven hundred and forty-two (74%) benign and 255 (26%) malignant masses were included. The IOTA models did better than the non-IOTA models (AUCs between 0.941 and 0.956 vs. 0.839 and 0.928). The difference in AUC between the best IOTA and the best non-IOTA model was 0.028 [95% confidence interval (CI), 0.011-0.044]. The AUC of the RMI was 0.911 (difference with the best IOTA model, 0.044; 95% CI, 0.024-0.064). The superior performance of the IOTA models was most pronounced in premenopausal patients but was also observed in postmenopausal patients. IOTA models were better able to detect stage I ovarian cancer.Conclusion: External validation shows that the IOTA models outperform other models, including the current reference test RMI, for discriminating between benign and malignant adnexal masses. Clin Cancer Res; 18(3); 815-25. Ó2011 AACR.

“…These results are promising, but external validation of the models in the hands of less experienced examiners remains to be done. (9) Logistic regression 0.10 (1) age, (2) ascites, (3) blood flow within a solid papillary projection, (4) maximal diameter of the solid component (bounded at 50 mm), (5) irregular internal cyst walls, and (6) acoustic shadows Bay MLP 11-2a (12) Artificial neural network 0.15 (1) age, (2) hormonal therapy, (3) ascites, (4) maximum diameter of the lesion, (5) irregular internal cyst wall, (6) color score, (7) blood flow within papillary projection, (8) number of papillary projection, (9) maximum diameter of solid component, (10) multilocular-solid tumor, and (11) solid tumor Bay MLP 11-2b (12) Artificial neural network 0.15 (1) age, (2) personal history of ovarian cancer, (3) pelvic pain during examination, (4) ascites, (5) maximum diameter of the lesion, (6) irregular internal cyst wall, (7) blood flow within papillary projection, (8) acoustic shadowing, (9) maximum diameter of solid component, (10) unilocular tumor, and (11) solid tumor Bay Perc 11 (12) Artificial neural network 0.15 (1) age, (2) personal history of ovarian cancer, (3) ascites, (4) maximum diameter of the lesion, (5) irregular internal cyst wall, (6) color score, (7) blood flow within papillary projection, (8) acoustic shadowing, (9) maximum diameter of solid component, (10) unilocular tumor, and (11) solid tumor LS-SVM lin (11) Support vector machine 0.15 (1) maximum diameter of the solid component, (2) maximum diameter of the ovary, (3) age, (4) color score 4, (5) presence of a multilocular-solid lesion, (6) ascites, (7) personal history of ovarian cancer, (8) blood flow within papillary projection, (9) acoustic shadows, (10) previous use of hormonal therapy, (11) irregular internal cyst walls, and (12) whether the tumor is suspected to be of ovarian origin LS-SVM rbf (11) Support vector machine 0.12 (1) maximum diameter of the solid component, (2) maximum diameter of the ovary, (3) age, (4) color score 4, (5) presence of a multilocular-solid lesion, (6) ascites, (7) personal history of ovarian cancer, (8) blood flow within a papillary projection, (9) acoustic shadows, (10) previous use of hormonal therapy, (11) irregular internal cyst walls, and (12) whether the tumor is suspected to be of ovarian origin LS-SVM add rbf (11) Support vector machine 0.12 (1) maximum diameter of the solid component, (2) maximum diameter of the ovary, (3) age, (4) color score 4, (5) presence of a multilocular-solid lesion, (6) ascites, (7) personal history of ovarian cancer, (8) blood flow within a papillary projection, (9) acoustic shadows, (10) previous use of hormonal therapy, (11) irregular internal cyst walls, and (12) whether the tumor is suspected to be of ovarian origin RVM lin (11) RVM 0.20 (1) maximum diameter of the solid component, (2) maximum diameter of the ovary, (3) age, (4) color score 4, (5) presence of a multilocular-solid lesion, (6) ascites, (7) personal history of ovarian cancer, (8) blood flow within a papillary projection, (9) acoustic shadows, (10) previous use of hormonal therapy, (11) irregular internal cyst walls, and (12) whether the tumor is suspected to be of ovarian origin RVM rbf (11) RVM 0.15 (1) maximum dia...…”

Section: Translational Relevancementioning

confidence: 99%

“…All models performed well when tested on the test set of patients (n = 312). Their area under the receiver operating characteristics (ROC) curve (AUC) ranged from 0.93 to 0.95 (9,11,12).…”

mentioning

confidence: 99%

Prospective Internal Validation of Mathematical Models to Predict Malignancy in Adnexal Masses: Results from the International Ovarian Tumor Analysis Study

Clinical Cancer Research

Testa

et al. 2009

Self Cite

Purpose: To prospectively test the mathematical models for calculation of the risk of malignancy in adnexal masses that were developed on the International OvarianTumorAnalysis (IOTA) phase 1data set on a new data set and to compare their performance with that of pattern recognition, our standard method. Methods: Three IOTA centers included 507 new patients who all underwent a transvaginal ultrasound using the standardized IOTA protocol. The outcome measure was the histologic classification of excised tissue. The diagnostic performance of 11 mathematical models that had been developed on the phase 1data set and of pattern recognition was expressed as area under the receiver operating characteristic curve (AUC) and as sensitivity and specificity when using the cutoffs recommended in the studies where the models had been created. For pattern recognition, an AUC was made based on level of diagnostic confidence. Results: All IOTA models performed very well and quite similarly, with sensitivity and specificity ranging between 92% and 96% and 74% and 84%, respectively, and AUCs between 0.945 and 0.950. A least squares support vector machine with linear kernel and a logistic regression model had the largest AUCs. For pattern recognition, the AUC was 0.963, sensitivity was 90.2%, and specificity was 92.9%. Conclusion: This internal validation of mathematical models to estimate the malignancy risk in adnexal tumors shows that the IOTA models had a diagnostic performance similar to that in the original data set. Pattern recognition used by an expert sonologist remains the best method, although the difference in performance between the best mathematical model is not large.