Testing non‐cognitive attributes in selection centres: how to avoid being reliably wrong

Selection methods in healthcare professions: where are we now and where next?

“…Selection research in healthcare has tended to focus largely on reliability (e.g. how many stations are required for an MMI to be reliable), here we remind readers that it is quite possible to be reliably wrong (Patterson and Ferguson 2012), and greater attention should be paid to establishing the predictive and construct validity of selection systems.…”

Development of Critical Thinking in Pharmacy Education

“…International Critical Thinking Test • It is a sensitive indicator of college student achievement, 27 and has demonstrated gains with a "college effect" 28 • Notably, it is also an indicator for development of a professional [29][30][31] • It is very similar to situational judgment tests used in admissions and hiring 32,33 a 'college effect.' 28 That is, the DIT score was higher in college graduates than those that did not attend college, was higher in college seniors compared with college freshman, and was higher in students that have had more intense college experiences than others.…”

“…cardiology, pulmonary, oncology, end-of-life care) as well as recognizing that often more than one "correct" answer exists, with better answers not being just memorized facts of logic but depending more on each case's situational context. Situational judgment tests are an application of complex thinking assessment focused towards professional program admissions and hiring; 32,33 they are very similar to the DIT.…”

“…54 With either study results, it would seem advantageous, compared to any small information gains, to change focus towards noncognitive assessments, such as building a much more reliable interview process. [55][56][57] This non-cognitive assessment approach is aligned with the 'holistic admissions' movement, 58 as well as at least the last two versions of PharmD standards.…”

Opening the black box of selection

“…For example, if the intended constructs are not measured and the predictive value is insufficient, the selection committee should go back to the drawing board, since the procedure is neither effective nor defensible or fair (Patterson and Zibarras 2018). Alternatively, if there is predictive value but the intended constructs are not measured, where the predictive value is coming from should be investigated in order to avoid 'being reliably wrong' (Patterson and Ferguson 2012) and measuring an unrelated construct that, by chance, correlates with study success (e.g. shoe-tying-skills could be predictive of medical school performance, but cannot defensibly be used as a selection tool).…”