Monte Carlo studies of bootstrap variability in ROC analysis with data dependency

Abstract

ROC analysis involving two large datasets is an important method for analyzing statistics of interest for decision making of a classifier in many disciplines. And data dependency due to multiple use of the same subjects exists ubiquitously in order to generate more samples because of limited resources. Hence, a two-layer data structure is constructed and the nonparametric twosample two-layer bootstrap is employed to estimate standard errors of statistics of interest derived from two sets of data, such as a weighted sum of two probabilities. In this article, to reduce the bootstrap variance and ensure the accuracy of computation, Monte Carlo studies of bootstrap variability were carried out to determine the appropriate number of bootstrap replications in ROC analysis with data dependency. It is suggested that with a tolerance 0.02 of the coefficient of variation, 2,000 bootstrap replications be appropriate under such circumstances.

Standard errors and significance testing in data analysis for testing classifiers

“…Indeed, to reduce the bootstrap variance and ensure the computation accuracy, our prior rigorous statistical research was carried out, such as the bootstrap variability studies that took months of CPU time to determine the appropriate number of bootstrap replications, the validation study by comparing the SEs of AUC estimated using the bootstrap algorithm on large i.i.d. datasets against those computed using the well-established analytical Mann-Whitney statistic method, using the multinomial probabilities to determine which bootstrap approach in the two-layer data structure should be used, and so on [4,[10][11][17][18][19][20][21][22].…”

“…In our ROC analysis for decision making of classifiers, data samples of scores are over tens of thousands and have no parametric model to fit, the statistics of interest are mostly probabilities or a weighted sum of probabilities, and data dependency may be involved. Thus, to reduce the bootstrap variance and ensure the computation accuracy, the bootstrap variabilities were re-studied, which took months of CPU time, and the appropriate number of bootstrap replications B under the above circumstances was determined to be 2,000 [11,[18][19][20][21][22].…”

“…where the number of iterations N is set to be 2,000, based on our extensive studies of bootstrap variability in ROC analysis with or without data dependency for large datasets [20][21][22].…”

“…In ROC analysis on large datasets, our prior rigorous statistical research was conducted, such as the validation study to provide a sound foundation for using the bootstrap method [17], the extensive bootstrap variability studies to determine the appropriate number of bootstrap replications without and with data dependency in order to reduce the bootstrap variance and ensure the computation accuracy, which took months of CPU time [11,[18][19][20][21][22], and so on. Our data structure and the corresponding bootstrap approach has sound scientific basis.…”

Diagnostic Accuracy with Total Adenosine Deaminase as a Biomarker for Discriminating Pleural Transudates and Exudates in a Population-Based Cohort Study

“…Here, a bootstrap confidence interval (1000 resamples; a random number seed of 978) was employed to generate a 95% CI of AUC. A tolerance 0.02 with 1000 replications was considered appropriate [27].…”