An evaluation of vignettes for predicting variation in the quality of preventive care

Abstract

Vignettes can measure and predict prevention performance. Vignettes may be a less costly way to assess prevention performance that also controls for patient case-mix.

Guideline-Inconsistent Breast Cancer Screening for Women over 50: A Vignette-Based Survey

“…[22][23][24][25] The survey used vignettes, which are accurate and validated tools for measuring preventive care practices. [26][27][28] The Women's Health Survey was sent to 3,200 physicians under age 65 who practiced in either an office or hospital based setting; it achieved a 61.7 % response rate. Using stratified random sampling, these physicians were selected from the 2008 American Medical Association (AMA) Physician Masterfile, a national list of all licensed physicians, in roughly equal numbers of family physicians, general internists, and obstetrician-gynecologists to ensure adequate numbers of respondents from each specialty type.…”

Methods of Observing Variations in Physicians’ Decisions: The Opportunities of Clinical Vignettes

“…By using the latter approaches, however, one can directly measure physician decisions and control for case mix by soliciting a decision on a single case or a consistent set of cases from all sampled providers (Table 1). [3][4][5][6][7] Medical Record Abstraction. Medical record abstraction relies on a trained chart abstractor to review clinical records and produce a data set of physician decisions as physicians themselves record them.…”

“…Both handwritten and electronic medical records also suffer from Brecording bias,^in that not all relevant medical data or services may be recorded. 3,5,6,8 Claims. As a record of physician point-of-care decisions, computerized administrative claims data share many of the advantages of medical record abstraction, being both widely available and requiring no provider time for data collection.…”

“…Yet, for many services (for example, advanced imaging), claims will only reliably identify the provider who is paid to perform the service, while the provider who decides to order the service (and the parameters of that decision) is of greater interest to policymakers. Claims also normally do not contain all the clinical data, 4 Manual abstraction is costly [3][4][5][6] and timeintensive 4,7 Sample sizes are likely to be small Medical records often lack detail and may omit decisions that occurred [3][4][5][6]8 and/or include decisions that never occurred 8 Only some EHR data fields can be automatically extracted (e.g., checkboxes, drop-down menus) 5 May be challenging to sufficiently control for differences in providers' patient case mixtures 3,4 Physicians who permit access to medical records may not be representative May be challenging to sufficiently control for differences in providers' patient case mixtures, 4 and bias in coding of claims may distort results (e.g., upcoding) 11 Often not possible to attribute a given service to the provider responsible for placing the order (e.g., an imaging study is likely to be attributed to a radiologist, rather than the primary care physician who submitted the order)…”

“…Able to capture clinical decisions that are not available through claims nor documented in the medical record By presenting a standardized patient case, controls for differences in physicians' case mixture 6 Hiring/training standardized patients is costly [3][4][5][6]12 Logistically challenging to organize visits of standardized patients 6 Sample sizes are likely to be small High burden for physicians 3,4 Treating standardized patients reduces physician time for actual patients 3 Not all decisions may be realistically simulated (e.g., pediatrics, signs of trauma) 7 Physicians who agree to participate in blinded standardized patient study may not be representative demand tests) during a clinical visit and to document the services they receive during the encounter. Like medical record abstraction, the use of standardized patients is presently valuable on a small scale, but likely unrealistic in large-scale studies of variations in point-of-care decisions across diverse communities and practice settings.…”