Identifying validity evidence for uncertainty tolerance scales: A systematic review

Abstract

PurposeUncertainty tolerance (UT) is increasingly valued as a medical graduate attribute and broadly measured among medical student populations. However, the validity evidence underpinning UT scale implementation has not been summarised across studies. The present work evaluates UT scale validity evidence to better inform when, why and how UT scales ought to be used and to identify remaining validity evidence gaps.MethodsA literature search for psychometric studies of UT scales was completed in 2022. Records were included if they implemented one of the four most cited UT scales (i.e. Physicians' Reactions to Uncertainty scale 1990 [PRU1990] or 1995 [PRU1995], Tolerance for Ambiguity [TFA] scale or Tolerance of Ambiguity in Medical Students and Doctors scale [TAMSAD]) in a population of physicians and/or medial students and presented validity evidence according to the Standards for Educational and Psychological Testing framework. Included studies were rated and analysed according to evidence for test content, response processes, internal structure, relations to other variables and consequences of testing.ResultsAmong the investigated scales, ‘relations to other variables’ and ‘internal structure’ were the most commonly reported forms of validity evidence. No evidence of ‘response processes’ or ‘consequences of testing’ was identified. Overall, the PRU1990 and PRU1995 demonstrated the strongest validity evidence, although evidence primarily related to physician populations.ConclusionsNone of the studied scales demonstrated evidence for all five sources of validity. Future research would benefit from assessing validity evidence for ‘response processes’ and ‘consequences of testing’ among physicians and medical students at different training/career stages to better understand UT construct conceptualisation in these populations. Until further and stronger validity evidence for UT scales is established, we caution against implementing UT scales outside of research settings (e.g. for higher stakes decision making).

In Reply to Hancock and Mattick

“…Crucially, and aligning with meta-analytic methods, 2 our article highlights the research gap concerning UT measurement among veterinary and medical students. We commend Hancock, Mattick, and their colleagues for developing the TAMSAD.…”

“…1 Fully exploring the development and validity testing of UT scales was the focus of our subsequent work. 2…”

How is “intolerance of uncertainty” (IU) measured? A systematic review of assessment tools for IU and the psychometric properties of IU questionnaires.

“…There have been systematic review studies on the psychometric properties of uncertainty tolerance scales specifically targeting physicians and medical students, who have to make clinical decisions when uncertain (Stephens et al, 2022(Stephens et al, , 2023. To date, however, there has been no systematic review of IU measures used with the general public or across a range of patient groups, and their psychometric properties and suitability for different purposes.…”

Uncertainty tolerance scales: Weighing up the research

“…In this issue, Stephens et al contribute to this growing body of evidence by presenting the validity evidence for four quantitative UT measurement scales. In doing so, they discuss when, why and how UT scales could be used to better effect 3 . To do so, the authors evaluated each using the five domains set out by Standards for Educational and Psychological Testing, 4 including test content, response process, internal structure, relations to other variables and consequences of testing.…”

“…According to broader validity frameworks, for selection tools and/or procedures to be considered to have merit, face validity, construct validity, cost-effectiveness and political validity should all be evaluated. 2,3 Applicant acceptability-a component of political validity-seems particularly important in this case given that applicants' lives and careers are directly affected by selection policies. What would the ones who actually have to undergo the selection procedure think when they fill out their application form and come across an inquiry about their athletic achievements?…”