Opening the black box of selection

Abstract

Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcomebased selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the setup of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.

COVID 19: Designing and conducting an online mini-multiple interview (MMI) in a dynamic landscape

“…After estimating costs (e.g. administrative, academic and IT time, software licences, equipment, refreshments for assessors, etc), we will be able to compare groups on outcome measures, such as examination performance, to examine what was spent in relation to the value that was returned (Schreurs et al 2018(Schreurs et al , 2019. This kind of cost-benefit analysis can inform decisions as to the optimal use of resources when planning selection in the future.…”

Outcomes-Based Selection Into Medical School: Predicting Excellence in Multiple Competencies During the Clinical Years

“…Prior research indicates that the admissions procedure is robust and replicable. 4 , 30 In another study, 30 we compared the constructs we intended to measure (i.e., the derived competencies) with the applicants’ results and found that the constructs we intended to measured were indeed the ones we measured. For more information on the specific selection procedure employed in the current study or on its development, content, and psychometric properties, please see our previous studies.…”

“…For more information on the specific selection procedure employed in the current study or on its development, content, and psychometric properties, please see our previous studies. 4 , 30 …”

Cognitive diagnostic modelling in healthcare professions education: an eye-opener

“…This property makes cognitive diagnosis modelling a feasible alternative approach that can encompass the multidimensionality known to exist within items, something intrinsic to certain well-known assessment tools, such as the situational judgement test (Garcia et al, 2014 ). The first instance of a peer-reviewed study using the cognitive diagnostic modelling approach in healthcare professions education was recently published in this journal (Schreurs et al, 2020 ). The authors looked for validity evidence based on the internal structure and response processes of two situational judgement tests used to select undergraduate medical students.…”

“…Two parameters are common to both the DINA and G-DINA models: the guessing parameter and the slipping parameter. Whereas the guessing parameter is a false-positive indicator, i.e., the probability of success when the examinee does not have the attributes, the slipping parameter can be seen as a false-negative indicator, i.e., the probability of failure when the examinee has the attributes (de la Torre & Minchen, 2014 ; Schreurs et al, 2020 ).…”