Variation in restorative treatment decisions: application of Receiver Operating Characteristic curve (ROC) analysis

Abstract

It has been evident for many years that dentists, when planning treatment for patients, do not act in a standard manner, and previous research has shown there to be wide variations in treatment planning amongst groups of dentists. Signal detection theory and Receiver Operating Characteristic (ROC) analysis allows measurement of an observer's ability to detect a lesion, while at the same time allowing examination of how a lesion, once perceived, is judged to be in need of treatment. An ROC curve is constructed by plotting the sensitivity (or true positive rate) of decisions made, against the false positive rate (equivalent to 1-specificity) when various decision attitudes, from interventionist to non-interventionist, are held. Fifteen pairs of simulated bitewing radiographs were shown to 20 dentists, who were asked to specify, for each approximal lesion, whether or not they would place a conventional restoration. The 7200 decisions made by the dentists were validated by sectioning and microscopically examining the teeth. The mean sensitivity of the dentists' decisions, when the strictest operating thresholds were held and caries into dentine was the validating criterion, was 0.26 and the mean specificity was 0.96. ROC analysis shows that when operating at the strictest threshold, the dentists were implying that specificity was weighted as being 2.7 times more important than sensitivity. ROC analysis leads to insight into how dentists differentially weight the true and false, positive and negative, outcomes of their decisions and thus allows explanation of why two dentists would rarely make exactly the same treatment plan for one patient, and also why different treatments might be offered to two patients exhibiting the same levels of disease.

Reducing variability in treatment decision-making: effectiveness of educating clinicians about uncertainty

“…The method is equivalent to repeatedly asking dentists to make a simple dichotomous decision whilst altering their attitudes towards interventive treatment (Hanley & McNeil 1982). The area beneath an ROC curve therefore estimates the probability that a radiographically evident lesion is treated as `more abnormal' by a dentist (or group of dentists) than a radiograph of a sound tooth (Kay & Knill‐Jones 1992). The areas under the ROC curves for each dentist and for all dentists in each of the three groups were derived using a computer program (Centor 1985).…”

“…Previous studies have shown considerable clinical variability in dental restorative treatment thresholds and decisions (Mileman et al 1982; Elderton & Nuttall 1983), and a previous study by Kay et al (1992) indicated that the key to achieving `good' clinical decisions lies in educating practitioners to understand the trade‐offs between false negative and false positive errors. Another study which focused on the reduction of clinical variability suggested that, if a clinician understands the principles underlying receiver operating characteristic curve analysis, his/her ability to make decisions consistent with his/her own attitudes to treatment might be improved (Hanley & McNeil 1982; Kay & Knill‐Jones 1992).…”

Conventional, Digital and Contrast–Enhanced Bitewing Radiographs in the Decision to Restore Approximal Carious Lesions

“…The investigators had to indicate on a 6-point confidence rating scale the certainty of their decision: 0 = definitely not, 1 = probably not, 2 = possibly not, 3 = possibly, 4 = probably, 5 = definitely. The characteristics of the imaginary patient were briefly described in accordance with a previously published patient's profile: 20-year-old patient, general medical anamnesis inconspicuous, average caries experience and oral hygiene level, readiness to observe regular recall dates over years to come [Kay and Knill Jones, 1992]. No information was given on lesion prevalence.…”

Concordance between responses to questionnaire scenarios and actual treatment to repair or replace dental restorations in the National Dental PBRN

“…64 The consistency within individual practitioners’ treatment approaches across primary occlusal caries, primary proximal caries, and secondary caries suggests that diagnostic and treatment differences across practitioners for specific restorations may reflect fundamental differences in dentists’ approaches to similar clinical findings. 16173 The lack of agreement over time will result in a specific patient receiving additional restorative treatment. Some of the additional treatment may be to teeth that will invariably require treatment, however, some will be to teeth that may remain stable over time, particularly in low caries risk populations.…”