Which Doctor? Combining Vignettes and Item Response to Measure Doctor Quality

Abstract

We develop a method in which vignettes-a battery of questions for hypothetical cases-are evaluated with item response theory to create a metric for doctor quality. The method allows a simultaneous estimation of quality and validation of the test instrument that can be used for further refinements. The method is applied to a sample of medical practitioners in Delhi, India. The method gives plausible results, rationalizes different perceptions of quality in the public and private sectors and pinpoints several serious problems with health care delivery in urban India. The findings confirm for instance, that the quality of private providers located in poorer areas of the city is significantly lower than those in richer neighborhoods. Surprisingly, similar results hold for providers in the public sector with important implications for inequities in the availability of health care.

The impact of recall periods on reported morbidity and health seeking behavior

“…As a result, we also variously use the total duration of health events in a month or condition on having at least one health event in a given month to ensure that the results on recall and morbidity are not driven by potential "doublecounting". 11 The term "doctors" has to be interpreted carefully here since the range of providers of medical services is very wide in India -from MD's and MBBS (standard medical university training) doctors to people knowledgeable of Indian traditional medicine (but prescribe allopathic medicine anyway) to genuine "quacks" with no training whatsoever (Das and Hammer, 2005). However, when we turn to the weekly data, an entirely different story emerges.…”

Defining Access to Health Care: Evidence on the Importance of Quality and Distance in Rural Tanzania

“…Although vignettes measure competence rather than actual performance (Leonard and Masatu 2005;Das and Hammer 2005), they are a valuable tool for comparing quality across providers because every clinician in the sample faces exactly the same patients. Observing clinicians in practice is a more accurate representation of the effort that clinicians exert with their regular patients, but it is more difficult to control for the different case mix across clinicians in the sample.…”

Strained Mercy: The Quality of Medical Care in Delhi

“…We ended up using 79 questions-those asked frequently enough to stand up to the statistical analysis-over the Note: This table provides examples of questions that the provider had to ask and treatments that she had to provide for a positive evaluation. Weights assigned to each question were assigned through the IRT methodology as described in the test and in greater detail in Das and Hammer (2003a). five cases. This method is related to other statistical techniques such as principal components analysis or factor analysis; an important advantage is that besides an overall score for providers, the method also provides a measure of how well each question performs in being able to distinguish between good and bad providers-how difficult the question is, how easy it is to guess the right answer, and how much difference there is between good and bad providers.…”

“…Thus while free medication will lead to greater use of the facility (equivalent to a pharmacist who "sells" his medicines free of cost), the impact on health outcomes depends entirely on the quality of care imparted. 3 This article summarizes findings in two recent papers: Hammer (2003a, and2003b) To answer these questions, we analyze two types of data collected from more than 200 medical care providers in seven neighborhoods of the city. 4 The first, on provider competence, is based on a set of "vignettes," or five hypothetical cases presented to providers who were asked what they would do in each circumstance-what questions they would ask, what examinations they would perform, and what treatments they would recommend.…”