The Reliability of Assessing the Appropriateness of Requested Diagnostic Tests

Abstract

Despite a poor reliability, peer assessment is the traditional method to assess the appropriateness of health care activities. This article describes the reliability of the human assessment of the appropriateness of diagnostic tests requests. The authors used a random selection of 1217 tests from 253 request forms submitted by general practitioners in the Maastricht region of The Netherlands. Three reviewers independently assessed the appropriateness of each requested test. Interrater kappa values ranged from 0.33 to 0.42, and kappa values of intrarater agreement ranged from 0.48 to 0.68. The joint reliability coefficient of the 3 reviewers was 0.66. This reliability is sufficient to review test ordering over a series of cases but is not sufficient to make case-by-case assessments. Sixteen reviewers are needed to obtain a joint reliability of 0.95. The authors conclude that there is substantial variation in assessment concerning what is an appropriately requested diagnostic test and that this feedback method is not reliable enough to make a case-by-case assessment. Computer support maybe beneficial to support and make the process of peer review more uniform.

Performing stable angina pectoris: An ethnographic study

“…Our findings suggest explanations for disagreements on angina diagnosis between physicians and data collected via standardized patient questionnaires (Friedman & Byington, 1985) and the high inter-rater variability in the interpretation of diagnostic tests (Bindels, Hasman, van Wersch, Pop, & Winkens, 2003). Although other studies have found gender differences in the presentation of angina pain (Philpott et al, 2001;Shaw et al, 2006), in this study the range of symptoms presented by women and the terms they used to describe them did not differ from those of men.…”

The assessment of interpretation in clinical biochemistry: a personal view

“…11 A recent paper identi¢ed both intra-and inter-assessor di¡erences, and suggested that a minimum of16 assessors was needed to arrive at a consistent (but not necessarily correct) verdict 95% of the time. 12 However, this extrapolation depends on an implicit assumption that the sole reason for intraand inter-assessor di¡erences is random variation, akin to intra-and inter-assay imprecision in conventional analytical EQA. This is obviously invalid since non-random variables exist between di¡erent assessors.…”

Evidence‐Based Imaging Guidelines and Medicare Payment Policy

“…The ACR Appropriateness Criteria were developed using the RAND methodology of generating appropriateness scores for medical tests and procedures, which combines a thorough literature review and synthesis with expert panel consensus development using a modified Delphi technique (Brown 1968;Dalkey 1969;Brook, Chassin, and Fink 1986;Phelps 1993;Brook 1994;Black 1998). The general conclusion of the large body of literature assessing the reliability and validity of appropriateness scores derived by the RAND method is that the method is reproducible and a valuable tool for generating clinical appropriateness criteria (Shekelle et al 1998(Shekelle et al , 2001Barnato and Garber 2003;Bindels et al 2003). The AHRQ National Guideline Clearinghouse (NGC) now includes structured abstracts of the ACR Appropriateness Criteria in its repository.…”