Combining independent studies of a diagnostic test into a summary roc curve: Data‐analytic approaches and some additional considerations

Abstract

We consider how to combine several independent studies of the same diagnostic test, where each study reports an estimated false positive rate (FPR) and an estimated true positive rate (TPR). We propose constructing a summary receiver operating characteristic (ROC) curve by the following steps. (i) Convert each FPR to its logistic transform U and each TPR to its logistic transform V after increasing each observed frequency by adding 1/2. (ii) For each study calculate D = V - U, which is the log odds ratio of TPR and FPR, and S = V + U, an implied function of test threshold; then plot each study's point (Si, Di). (iii) Fit a robust-resistant regression line to these points (or an equally weighted least-squares regression line), with V - U as the dependent variable. (iv) Back-transform the line to ROC space. To avoid model-dependent extrapolation from irrelevant regions of ROC space we propose defining a priori a value of FPR so large that the test simply would not be used at that FPR, and a value of TPR so low that the test would not be used at that TPR. Then (a) only data points lying in the thus defined north-west rectangle of the unit square are used in the data analysis, and (b) the estimated summary ROC is depicted only within that subregion of the unit square. We illustrate the methods using simulated and real data sets, and we point to ways of comparing different tests and of taking into account the effects of covariates.

Renovascular disease in patients with hypertension: detection with duplex ultrasound

“…the marks representing the different studies and ultrasound strategies. 40 For quantitative analysis of the literature data the diagnostic Odds Ratio (= sensitivity × specificity/{1 − sensitivity} × {1 − specificity}) was calculated for each study (Table 4a-c). The Spearman rank correlation coefficients between diagnostic Odds Ratio's of the separate studies and a number of study variables, ie number of patients, prevalence of renovascular disease, year of publication, are less than 0.3, indicating a lack of influence of these variables on the diagnostic performance of duplex ultrasound.…”

Diagnosing Suspected Scaphoid Fractures: A Systematic Review and Meta-analysis

“…To prevent division by 0 when pooling the ln DOR, conventional correction was applied by adding 0.5 to each cell when the 2 9 2 table for a study contained one or more 0 values. The diagnostic performance of each test also was assessed using SROC curves, which allow for a trend in DOR with threshold [36]. These curves can be defined by a regression model: D = a + bS, where D is ln DOR and S, a measure of the diagnostic threshold, is equivalent to the sum of logits of the true-positive rate and the false-positive rate.…”

Bayesian meta-analysis of Papanicolaou smear accuracy

“…The summary receiver operating characteristic (SROC) method, introduced by Littenberg and Moses [2,3], has been widely used by authors of published meta-analyses that examine diagnostic test accuracy. However, the SROC method has some inherent limitations that could affect its validity and usefulness.…”

“…However, the SROC method has some inherent limitations that could affect its validity and usefulness. First, the SROC method is not based on a formal probability model; in fact, Littenberg and Moses call it a "data-analytic approach" [2]. Second, there are difficulties in dealing with any zeroes that appear in any cells of the 2×2 table used in diagnostic test evaluation, i.e.…”

“…true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). The continuity correction method, i.e., adding one half to each count in calculating the true-positive rate and the false-positive rate, introduces non-negligible downward bias to the estimated SROC curve [2,3,4]. An additional problem with the SROC method is that confidence intervals are based on large sample theory approximations and the appropriate sample size for a meta-analysis is the number of studies which may not be large enough for the approximation to be accurate.…”