Evaluating Diagnostic Tests with Imperfect Standards

Abstract

New diagnostic tests frequently are evaluated against gold standards that are assumed to classify patients with unerring accuracy according to the presence or absence of disease. In practice, gold standards rarely are perfect predictors of disease and tend to misclassify a small number of patients. When an imperfect standard is used to evaluate a diagnostic test, many commonly used measures of test performance are distorted. It is not widely appreciated that these distortions occur in predictable directions and that they may be of considerable magnitude, even when the gold standard has a high degree of accuracy. The diagnostic powers of clinical tests will be more accurately reported if consideration is given to the types of biases that result from the use of imperfect standards. Several different approaches may be used to minimize these distortions when evaluating new tests.

How to diagnose rheumatoid arthritis early: A prediction model for persistent (erosive) arthritis

“…This discrepancy is explained by the inevitable occurrence of circularity, resulting in overestimation of the diagnostic properties of the criteria when the clinical diagnosis is used as the gold standard (28). Defining the gold standard in terms of arthritis outcome prevents the occurrence of circularity (10). Moreover, the outcome categories used in this study represent clearly defined, objective, biologic diagnoses: arthritis and erosions.…”

“…This leads to circularity and overestimation of the diagnostic properties of these tests (9). Defining the gold standard in terms of clinical outcome is a way of avoiding this circularity (10). Moreover, predicting the outcome of arthritis in a patient is more relevant for therapeutic decision-making than predicting whether an arthritis syndrome will ever satisfy a set of classification criteria.…”

Autoantibodies to cyclic citrullinated peptides predict progression to rheumatoid arthritis in patients with undifferentiated arthritis: A prospective cohort study

“…Since there is no independent standard or test for RA, the ACR criteria are widely used as the "gold standard." However, using an imperfect standard to evaluate a new diagnostic test is not ideal (19). One possible solution is to frame the diagnostic problem in terms of clinical outcome instead of using the ACR criteria.…”

The true treatment benefit is unpredictable in clinical trials using surrogate outcome measured with diagnostic tests

“…When both tests are evaluated in the same population, the baseline risk is obtained from: r 0 = (prev (se 1 × se 2 ) + ((1-prev)(1-sp 1 )(1-sp 2 ))), where se 1 and sp 1 are sensitivity and specificity of the reference standard and se 2 and sp 2 are true sensitivity and specificity of the new test [23]. When errors between the new and the reference tests are correlated (occur in the same patients) the baseline risk is obtained from: r 0 =(prev ((se 1 …”

“…Errors in two tests are independent when they do not occur in the same patients, and are correlated when both tests misclassify the same patients. Figures 5 and 6 show that the observed sensitivity and specificity of a new test compared to an imperfect reference is underestimated when errors are independent, and could be over-or underestimated when errors are correlated [23,62,63].…”