Some issues in resolution of diagnostic tests using an imperfect gold standard

Abstract

As a subject's true disease status is seldom known with certainty, it is necessary to compare the performance of new diagnostic tests with those of a currently accepted but imperfect 'gold standard'. Errors made by the gold standard mean that the sensitivity and specificity calculated for the new test are biased, and do not correctly estimate the new method's sensitivity and specificity. The traditional approach to this problem was 'discrepant resolution', in which the subjects for whom the two methods disagreed were subjected to a third 'resolver' test. Recent work has pointed out that this does not automatically solve the problem. A sounder approach goes beyond the discordant test results and tests at least some of the subjects with concordant results with the resolver also. This leaves some issues unresolved. One is the basic question of the direction of biases in various estimators. We point out that this question does not have a simple universal answer. Another issue, if one is to test a sample of the subjects with concordant results rather than all cases, is how to compute estimates and standard errors of the measures of test performance, notably sensitivity and specificity of the test method relative to the resolver. Expressions for these standard errors are given and illustrated with a numeric example. It is shown that using just a sample of subjects with concordant results may lead to great savings in assays. The design issue of how many concordant cells to test depends on the numbers of concordants and discordants. The formulae given show how to evaluate impact of different choices for these numbers and hence settle on a design that gives the required precision of estimates.

Seeking a Valid Gold Standard for an Innovative, Dialect-Neutral Language Test

“…Because the PPP of the CSaR was 0.55, we had some support for moving half of the Dlv+/CS+ to Cell a: true positives. We grant, as argued by critics of discrepant resolution (summarized in Hawkins et al, 2001), that we were ignoring the cases of CSaR/DELV-NR disagreement in Cells a and d, but our knowledge of the CSaR's diagnostic accuracy did not give us confidence to have it override two opposite diagnoses.…”

“…Thus, a refinement of the CSaR's diagnostic accuracy could give an indication of how much more confidence in DELV-NR scores is warranted when they agreed with CSaR in general, including the discrepant cases in which the DELV-NR disagreed with the LS, a procedure called discrepant resolution (Hawkins et al, 2001). Therefore, as shown in Table 5, we compared the DELV-NR outcomes to those of CSaR.…”

“…First, specificity and NPP were adequate, so one can have a fair amount of confidence in its diagnoses of typical development and will also expect between one quarter and two-thirds of the CSaR diagnoses of LI to be correct (95% CI around PPP [0.27, 0.68]). Given the large number of diagnoses expected to be incorrect, we would not have enough confidence in a CSaR diagnosis to override a DELV-NR diagnosis when there is no indication that the DELV-NR is wrong (as many critics of discrepancy resolution require; Hawkins et al, 2001). However, when the diagnosis of the DELV-NR has been put in doubt by an opposite diagnosis from the LS, we suggest that one is justified in accepting a CSaR diagnosis as a corroboration of the DELV-NR in proportion to the values of its predictive power calculated relative to LSs, as is illustrated in Table 5.…”

“…For medical decision making, where the disease status of individuals is seldom certain and available reference standards are often imperfect, several strategies have been proposed to combine data from two reference standards, using a second imperfect test as a "resolver test" in ways that attempt to compensate for potential biases in each of them (Green, Black, & Johnson, 1998;Hawkins, Garrett, & Stephenson, 2001).…”

Screening test accuracy studies: how valid are our conclusions? Application to visual inspection methods for cervical screening

“…An imperfect gold standard results in inaccurate estimates of sensitivity and specificity. Determining the magnitude of inaccuracy in the gold standard measure is not an easy task [3]. When no gold standard is really available, model-based approaches can be used like latent class analysis (LCA), but this requires tedious statistical analysis [4].…”