Thresholds and accuracy in screening tools for early detection of psychopathology

Abstract

Background The accuracy of any screening instrument designed to detect psychopathology among children is ideally assessed through rigorous comparison to ‘gold standard’ tests and interviews. Such comparisons typically yield estimates of what we refer to as ‘standard indices of diagnostic accuracy,’ including sensitivity, specificity, positive predictive value (PPV) and negative predictive value. However, whereas these statistics were originally designed to detect binary signals (e.g., diagnosis present or absent), screening questionnaires commonly used in psychology, psychiatry, and pediatrics typically result in ordinal scores. Thus, a threshold or ‘cut score’ must be applied to these ordinal scores before accuracy can be evaluated using such standard indices. To better understand the tradeoffs inherent in choosing a particular threshold, we discuss the concept of ‘threshold probability.’ In contrast to PPV, which reflects the probability that a child whose score falls at or above the screening threshold has the condition of interest, threshold probability refers specifically to the likelihood that a child whose score is equal to a particular screening threshold has the condition of interest. Method The diagnostic accuracy and threshold probability of two well-validated behavioral assessment instruments, the Child Behavior Checklist Total Problem Scale and the Strengths and Difficulties Questionnaire total scale were examined in relation to a structured psychiatric interview in three de-identified datasets. Results Although both screening measures were effective in identifying groups of children at elevated risk for psychopathology in all samples (odds ratios ranged from 5.2 to 9.7), children who scored at or near the clinical thresholds that optimized sensitivity and specificity were unlikely to meet criteria for psychopathology on gold standard interviews. Conclusions Our results are consistent with the view that screening instruments should be interpreted probabilistically, with attention to where along the continuum of positive scores an individual falls.

Screening for pickiness – a validation study

“…The majority of cases were detected (approximately 80%) with the suggested cut-offs of 3.00 for moderate and severe cases combined, and 3.33 for severe cases only; and rates of false negatives were low. However, high sensitivity is commonly accompanied with comparatively low PPV [9]: i.e. many children are typically falsely screened as positive.…”

“…Taylor et al [2] conclude that validation of more reliable measures of pickiness are needed [1]. To be considered valid the instrument in question must display acceptable levels of accuracy, which can be ensured by comparisons to ‘gold standard’ tests or psychiatric interviews [9]. However, to the best of our knowledge, no earlier study has examined screening for picky eating.…”

Is a Positive Developmental-Behavioral Screening Score Sufficient to Justify Referral? A Review of Evidence and Theory

“…7 A recent study that evaluated threshold probability for 2 well validated behavioral screening instruments (the Strengths and Difficulties Questionnaire and the Childhood Behavior Checklist) reported threshold probabilities ranging from 9% to 54%, despite high prevalence in 2 of the 3 samples considered (35% and 43% vs 14%). 7 These estimates of threshold probability were as low as a third of corresponding estimates of PPV.…”

“…7 A recent study that evaluated threshold probability for 2 well validated behavioral screening instruments (the Strengths and Difficulties Questionnaire and the Childhood Behavior Checklist) reported threshold probabilities ranging from 9% to 54%, despite high prevalence in 2 of the 3 samples considered (35% and 43% vs 14%). 7 These estimates of threshold probability were as low as a third of corresponding estimates of PPV. Because threshold probability is dependent on prevalence (just like PPV), the high base rates reported in the studies cited might overestimate the probability that children from a general population who score at the threshold would truly have developmental-behavioral disabilities.…”

“…If pediatricians respond by not referring most children who score just above the threshold, then they are effectively raising the screening threshold, thereby favoring predictive value at the cost of sensitivity. 7 For screening to be effective, understanding the tradeoffs in any threshold is essential.…”