Estimation of diagnostic test accuracy: A “Rule of Three” for data with repeated observations but without a gold standard

Abstract

This article considers how to estimate the accuracy of a diagnostic test when there are repeated observations, but without the availability of a gold standard or reference test. We identify conditions under which the structure of the observed data is rich enough to provide sufficient degrees of freedom, such that a suitable latent class model can be fitted with identifiable accuracy parameters. We show that a Rule of Three applies, specifying that accuracy can be evaluated as long as there are at least three observations per individual with the given test. This rule also applies if the three observations arise from combinations of different test methods, or from a sequential design in which individuals are tested for a maximum number of times with the same test but stopping if a positive (or negative) result occurs. The rule pertains to tests having an arbitrary number of response categories. Accuracy is evaluated by parameters reflecting rates of misclassification among the response categories, and the model also provides estimates of the underlying distribution of the true disease state. These ideas are illustrated by data from two medical studies. Issues discussed include the advantages and disadvantages of analyzing the response variable as binary or multinomial, as well as the feasibility of testing goodness of fit when the model incorporates a large number of parameters. Comparisons are possible between models that do or do not assume equal accuracy rates for the observations, and between models where certain misclassification parameters are or are not assumed to be zero.

Global and Local Assessment of Image Classification Quality on an Overall and Per-Class Basis without Ground Reference Data

“…However, this could also result in many cells in the contingency table being empty which can complicate the estimation and interpretation of model fit [36,37]. There is an extensive literature on latent class modelling, e.g., [32][33][34][35][36][37][38][39][40][41] and here only key details are provided for brevity.…”

“…The latent class analysis requires that a set of at least three image classifications are available as indicators of the unobserved (latent) variable and its classes [38,39]. In the analysis, the observed associations among the set of classification outputs generated is taken to be explainable by a latent variable that represents the thematic classes of interest [35,39].…”

“…The latent class analysis requires that a set of at least three image classifications are available as indicators of the unobserved (latent) variable and its classes [38,39]. In the analysis, the observed associations among the set of classification outputs generated is taken to be explainable by a latent variable that represents the thematic classes of interest [35,39]. The patterns of class labelling observed for the image pixels classified multiple times by the set of classifiers adopted may be used to form a latent class model from which the accuracy of each individual classification may be estimated without the use of ground reference data.…”