Introduction

Abstract

Medical informatics systems are often designed to perform at the level of human experts. Evaluation of the performance of these systems is often constrained by lack of reference standards, either because the appropriate response is not known or because no simple appropriate response exists. Even when performance can be assessed, it is not always clear whether the performance is sufficient or reasonable. These challenges can be addressed if an evaluator enlists the help of clinical domain experts. 1) The experts can carry out the same tasks as the system, and then their responses can be combined to generate a reference standard. 2)The experts can judge the appropriateness of system output directly. 3) The experts can serve as comparison subjects with which the system can be compared. These are separate roles that have different implications for study design, metrics, and issues of reliability and validity. Diagrams help delineate the roles of experts in complex study designs.

Defining the correctness of a diagnosis: differential judgments and expert knowledge

“…Alternatively, experts may be asked to perform a diagnostic task in order to set a gold standard relative to which others will be measured or asked to judge the performance of others (McGaghie et al 1994;Dillon et al 2002Dillon et al , 2004Berner 2003;Boulet et al 2003;Littlefield et al 2003;Ramnarayan et al 2003). However, a significant issue that arises from expert-based evaluation is inconsistency due to variations in practice, differences in judgment, and differences in degrees of harshness or lenience (Hills 1976;Hripcsak and Wilcox 2002;Takayama et al 2006;Iramaneerat and Yudkowsky 2007;Norcini and McKinley 2007). Additionally, measures of diagnostic accuracy have been developed by investigators who are interested in assessing the performance of diagnostic decision support systems (Friedman et al 1998;Berner 2003;Ramnarayan et al 2003).…”

“…In fact, definitions of diagnostic quality and accuracy are typically based on the knowledge and judgment of an individual expert or of a small panel of experts (Hripcsak and Wilcox 2002). Alternatively, experts may be asked to perform a diagnostic task in order to set a gold standard relative to which others will be measured or asked to judge the performance of others (McGaghie et al 1994;Dillon et al 2002Dillon et al , 2004Berner 2003;Boulet et al 2003;Littlefield et al 2003;Ramnarayan et al 2003).…”