Have admissions committees considered all the evidence?

Hecker, Kent G.; Norman, Geoff

doi:10.1007/s10459-016-9750-1

Cited by 13 publications

(15 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All in all, we have shown that an outcome-based, holistic selection procedure is predictive of study success across a variety of cognitive, (inter)personal skills and mixed assessments. Although we did not carry out direct comparisons with other tools, our outcome-based approach seems to address some of the limitations of individual selection tools in relation to predictive validity [ 7 , 10 , 13 , 15 , 43 ]. We urge others to consider designing and implementing outcome-based selection aligned with curricula and assessment processes, and encourage robust evaluations of the predictive validity of this approach in other contexts, as well as throughout the clinical years and beyond.…”

Section: Discussionmentioning

confidence: 99%

“…Developing a selection procedure that can fairly and accurately discriminate between applicants, based on academic as well as (inter)personal criteria, is challenging [ 10 – 13 ]. Many schools struggle with the question of what combination of tools to use to ensure that all desirable academic and (inter)personal qualities are assessed [ 14 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Selection into medicine: the predictive validity of an outcome-based procedure

et al. 2018

View full text Add to dashboard Cite

BackgroundMedical schools must select students from a large pool of well-qualified applicants. A challenging issue set forward in the broader literature is that of which cognitive and (inter)personal qualities should be measured to predict diverse later performance. To address this gap, we designed a ‘backward chaining’ approach to selection, based on the competences of a ‘good doctor’. Our aim was to examine if this outcome-based selection procedure was predictive of study success in a medical bachelor program.MethodsWe designed a multi-tool selection procedure, blueprinted to the CanMEDS competency framework. The relationship between performance at selection and later study success across a three-year bachelor program was examined in three cohorts. Study results were compared between selection-positive and selection-negative (i.e. primarily rejected) students.ResultsSelection-positive students outperformed their selection-negative counterparts throughout the entire bachelor program on assessments measuring cognitive (e.g. written exams), (inter)personal and combined outcomes (i.e. OSCEs). Of the 30 outcome variables, selection-positive students scored significantly higher in 11 cases. Fifteen other, non-significant between-group differences were also in favor of the selection-positives. An overall comparison using a sign test indicated a significant difference between both groups (p < 0.001), despite equal pre-university GPAs.ConclusionsThe use of an outcome-based selection approach seems to address some of the predictive validity limitations of commonly-used selection tools. Selection-positive students significantly outperformed their selection-negative counterparts across a range of cognitive, (inter)personal, and mixed outcomes throughout the entire three-year bachelor in medicine.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Selection into medicine: the predictive validity of an outcome-based procedure

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Third, even assuming that the number of non-academic attributes could be reduced to a manageable number, it is uncertain what each of the presently used tests of nonacademic attributes assesses. It has been claimed "tests like the MMI are less tests of a specific characteristic and more a test format adaptable to assessing many different attributes [which] regrettably leaves the specific attribute in the eye of the beholder-the author of the specific test" [34].…”

Section: Assessment Of Non-academic Attributes: Limitationsmentioning

confidence: 99%

Assessments of Non-academic Attributes in Applicants for Undergraduate Medical Education: an Overview of Advantages and Limitations

Benbassat

2019

Med.Sci.Educ.

View full text Add to dashboard Cite

“…Christian et al 2010;Hecker and Norman 2017;Kreiter 2017;Kulasegaram 2017;Patterson et al 2017) in order to determine whether the intended constructs are measured. Research on this subject is sorely missing (Hecker and Norman 2017;Kulasegaram 2017), and would not only greatly benefit the defensibility of selection procedures (Kreiter 2017), but would also be a first step in the direction of creating more theory-based selection procedures (Patterson et al 2018;Prideaux et al 2011). Moreover, conducting studies on construct validity yields practical implications for selection.…”

Section: Introductionmentioning

confidence: 99%

Opening the black box of selection

Schreurs

Cleutjens

Collares

et al. 2019

Adv in Health Sci Educ

View full text Add to dashboard Cite

Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcomebased selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the setup of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.

show abstract

Have admissions committees considered all the evidence?

Cited by 13 publications

References 15 publications

Selection into medicine: the predictive validity of an outcome-based procedure

Selection into medicine: the predictive validity of an outcome-based procedure

Assessments of Non-academic Attributes in Applicants for Undergraduate Medical Education: an Overview of Advantages and Limitations

Opening the black box of selection

Contact Info

Product

Resources

About