The social validity of a national assessment centre for selection into general practice training

Abstract

BackgroundInternationally, recruiting the best candidates is central to the success of postgraduate training programs and the quality of the medical workforce. So far there has been little theoretically informed research considering selection systems from the perspective of the candidates. We explored candidates’ perception of the fairness of a National Assessment Centre (NAC) approach for selection into Australian general practice training, where candidates were assessed by a Multiple Mini Interview (MMI) and a written Situational Judgment Test (SJT), for suitability to undertake general practice (GP) training.MethodsIn 2013, 1,930 medical practitioners, who were eligible to work in Australia attended one of 14 NACs in each of 5 states and 2 territories. A survey was distributed to each candidate at the conclusion of their assessment, which included open-ended questions aimed at eliciting candidates’ perceived benefits and challenges of the selection process. A framework analysis was informed by the theoretical lens of Social Validity Theory.ResultsQualitative data was available from 46% (n = 886/1,930) of candidates, who found the NAC experience fair and informative for their training and career goals, but wanted to be provided with more information in preparation. Candidates valued being able to communicate their skills during the MMI, but found some difficulty in interpreting the questions. A significant minority had concerns that a lack of relevant GP experience may inhibit their performance. Candidates also expressed concerns about the time limits within the written paper, particularly if English was not their first language. They also expressed a desire for formative feedback during the interview process.ConclusionDuring any job selection process, not only is the organisation assessing the candidates, but the candidates are also assessing the organisation. However, a focus on the candidate experience throughout an organisation’s selection process may provide benefits to both candidates and the organisation, regardless of whether or not candidates secured the job. Social Validity Theory is a useful addition to the methods for demonstrating the reasonableness of any selection system.

Utility of selection methods for specialist medical training: A BEME (best evidence medical education) systematic review: BEME guide no. 45

“…However, only a third agreed that the test gave them sufficient opportunity to indicate their ability for training and that the test would help selectors differentiate between applicants (Koczwara et al 2012). In a qualitative study focusing on the social validity of selection processes in the Australian GP setting, although overall rating for the combination of the MMI and the SJT was positive, there were concerns about the acceptability of the SJT by a small minority of the sample (18%) (Burgess et al 2014).…”

“…One article (Patterson et al 2014) was a qualitative study exploring competency models to improve uniformity and calibration of the overall process. Two articles described the Australian GP selection center process, two quantitatively (Roberts et al 2014;Patterson, Rowett, et al 2016) and one qualitatively (Burgess et al 2014), two describing a selection center approach into anesthetics training (Gale et al 2010;Roberts et al 2013), three describing the UK GP selection center approach (Mitchison 2009;Patterson, Baron, et al 2009;Patterson, Lievens, et al 2013) and a systematic review (Patterson, Knight, et al 2016).…”

“…For example, a recent development in validity research is around consequential validity, which describes the intended and unintended effects on stakeholders of any assessment (Cook et al 2015). Of the studies that addressing aspects of consequential validity of selection methods, one referred to the concept of political validity, and the other referred to social validity (Burgess et al 2014) in exploring candidates' perceptions of job relevance and overall fairness of the selection process in general practice training.…”

2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions

“…On the other hand, MMIs offer a face-to-face encounter allowing the perspectives of a preselected group of applicants to be probed in more depth. Face-to-face encounters may offer additional gains unrelated to the psychometric properties of the selection method: for example, they indicate institutional valuing of personal connections and visiting the medical school or specialty training setting which help applicants to assess the fit between themselves and the learning environment (Burgess, Roberts, Clark, & Mossman, 2014).…”

Multiple mini interview (MMI) for general practice training selection in Australia: interviewers’ motivation

“…Multiple Mini Interviews (MMIs) are being used by a growing number of postgraduate training programs and medical schools as their interview process to select entry into their training programs. Canada, Australia and the United Kingdom have used MMI to assess non-cognitive characteristics of postgraduate medical trainees, with early findings suggesting that the MMI offers a useful format to select junior doctors for specialty training [ 5 – 9 ]. The expectation is that this method will assist in the selection of candidates who are team players, with communication skills that allow them to connect with both patients and other healthcare professionals.…”