Why assessment in medical education needs a solid foundation in modern test theory

Abstract

Despite the frequent use of state-of-the-art psychometric models in the field of medical education, there is a growing body of literature that questions their usefulness in the assessment of medical competence. Essentially, a number of authors raised doubt about the appropriateness of psychometric models as a guiding framework to secure and refine current approaches to the assessment of medical competence. In addition, an intriguing phenomenon known as case specificity is specific to the controversy on the use of psychometric models for the assessment of medical competence. Broadly speaking, case specificity is the finding of instability of performances across clinical cases, tasks, or problems. As stability of performances is, generally speaking, a central assumption in psychometric models, case specificity may limit their applicability. This has probably fueled critiques of the field of psychometrics with a substantial amount of potential empirical evidence. This article aimed to explain the fundamental ideas employed in psychometric theory, and how they might be problematic in the context of assessing medical competence. We further aimed to show why and how some critiques do not hold for the field of psychometrics as a whole, but rather only for specific psychometric approaches. Hence, we highlight approaches that, from our perspective, seem to offer promising possibilities when applied in the assessment of medical competence. In conclusion, we advocate for a more differentiated view on psychometric models and their usage.

Clinical reasoning performance assessment: using situated cognition theory as a conceptual framework

“…From a psychometric perspective, the large unexplained variance in clinical reasoning performance may relate to at least three possibilities: (1) unmeasured factors, or (2) a more complex construct than presumed or (3) truly random statistical variance [19]. Statistics are agnostic about the possibility of unmeasured factors and construct complexity.…”

A situated cognition model for clinical reasoning performance assessment: a narrative review

“…This finding, named "context specificity", undoubtedly plays a role in the current crisis in medical error in the US [5,6]. The mechanisms which lead to context specificity remain uncertain [7][8][9]. Traditionally, information processing theories [10], which emphasize knowledge acquisition and organization, have provided valuable insights into understanding clinical reasoning, but these do not naturally lend themselves to the exploration of context.…”

In defence of constructivist, utility‐driven psychometrics for the ‘post‐psychometric era’

“…Ten Cate and Regehr remind us that one of the drivers of the psychometric mindset was to achieve fairness; this remains important and, hence, post‐psychometric approaches should not lead to pre‐psychometric mindsets . There is an important baby in the bathwater, one which may be opaque due to misconceptions regarding the term ‘psychometrics’ itself. What, if anything, can be salvaged from the psychometric scrapheap?…”