BackgroundEntry into specialty training was determined by a National Assessment Centre (NAC) approach using a combination of a behavioural Multiple-Mini-Interview (MMI) and a written Situational Judgement Test (SJT). We wanted to know if interviewers could make reliable and valid decisions about the non-cognitive characteristics of candidates with the purpose of selecting them into general practice specialty training using the MMI. Second, we explored the concurrent validity of the MMI with the SJT.MethodsA variance components analysis estimated the reliability and sources of measurement error. Further modelling estimated the optimal configurations for future MMI iterations. We calculated the relationship of the MMI with the SJT.ResultsData were available from 1382 candidates, 254 interviewers, six MMI questions, five alternate forms of a 50-item SJT, and 11 assessment centres. For a single MMI question and one assessor, 28% of the variance between scores was due to candidate-to-candidate variation. Interviewer subjectivity, in particular the varying views that interviewer had for particular candidates accounted for 40% of the variance in scores. The generalisability co-efficient for a six question MMI was 0.7; to achieve 0.8 would require ten questions. A disattenuated correlation with the SJT (r = 0.35), and in particular a raw score correlation with the subdomain related to clinical knowledge (r = 0.25) demonstrated evidence for construct and concurrent validity. Less than two per cent of candidates would have failed the MMI.ConclusionThe MMI is a moderately reliable method of assessment in the context of a National Assessment Centre approach. The largest source of error relates to aspects of interviewer subjectivity, suggesting enhanced interviewer training would be beneficial. MMIs need to be sufficiently long for precise comparison for ranking purposes. In order to justify long term sustainable use of the MMI in a postgraduate assessment centre approach, more theoretical work is required to understand how written and performance based test of non-cognitive attributes can be combined, in a way that achieves acceptable generalizability, and has validity.

BackgroundInternationally, recruiting the best candidates is central to the success of postgraduate training programs and the quality of the medical workforce. So far there has been little theoretically informed research considering selection systems from the perspective of the candidates. We explored candidates’ perception of the fairness of a National Assessment Centre (NAC) approach for selection into Australian general practice training, where candidates were assessed by a Multiple Mini Interview (MMI) and a written Situational Judgment Test (SJT), for suitability to undertake general practice (GP) training.MethodsIn 2013, 1,930 medical practitioners, who were eligible to work in Australia attended one of 14 NACs in each of 5 states and 2 territories. A survey was distributed to each candidate at the conclusion of their assessment, which included open-ended questions aimed at eliciting candidates’ perceived benefits and challenges of the selection process. A framework analysis was informed by the theoretical lens of Social Validity Theory.ResultsQualitative data was available from 46% (n = 886/1,930) of candidates, who found the NAC experience fair and informative for their training and career goals, but wanted to be provided with more information in preparation. Candidates valued being able to communicate their skills during the MMI, but found some difficulty in interpreting the questions. A significant minority had concerns that a lack of relevant GP experience may inhibit their performance. Candidates also expressed concerns about the time limits within the written paper, particularly if English was not their first language. They also expressed a desire for formative feedback during the interview process.ConclusionDuring any job selection process, not only is the organisation assessing the candidates, but the candidates are also assessing the organisation. However, a focus on the candidate experience throughout an organisation’s selection process may provide benefits to both candidates and the organisation, regardless of whether or not candidates secured the job. Social Validity Theory is a useful addition to the methods for demonstrating the reasonableness of any selection system.

BackgroundMultiple Mini Interviews (MMIs) are being used by a growing number of postgraduate training programs and medical schools as their interview process for selection entry. The Australian General Practice and Training (AGPT) used a National Assessment Centre (NAC) approach to selection into General Practice (GP) Training, which include MMIs. Interviewing is a resource intensive process, and implementation of the MMI requires a large number of interviewers, with a number of candidates being interviewed simultaneously. In 2015, 308 interviewers participated in the MMI process – a decrease from 340 interviewers in 2014, and 310 in 2013. At the same time, the number of applicants has steadily increased, with 1930 applications received in 2013; 2254 in 2014; and 2360 in 2015. This has raised concerns regarding the increasing recruitment needs, and the need to retain interviewers for subsequent years of MMIs. In order to investigate interviewers’ reasons for participating in MMIs, we utilised self-determination theory (SDT) to consider interviewers’ motivation to take part in MMIs at national selection centres.MethodsIn 2015, 308 interviewers were recruited from 17 Regional Training Providers (RTPs) to participate in the MMI process at one of 15 NACs. For this study, a convenience sample of NAC sites was used. Forty interviewers were interviewed (n = 40; 40/308 = 13%) from five NACs. Framework analysis was used to code and categorise data into themes.ResultsInterviewers’ motivation to take part as interviewers were largely related to their sense of duty, their desire to contribute their expertise to the process, and their desire to have input into selection of GP Registrars; a sense of duty to their profession; and an opportunity to meet with colleagues and future trainees. Interviewers also highlighted factors hindering motivation, which sometimes included the large number of candidates seen in one day.ConclusionInterviewers’ motivation for contributing to the MMIs was largely related to their desire to contribute to their profession, and ultimately improve future patient care. Interviewers recognised the importance of interviewing, and felt their individual roles made a crucial contribution to the profession of general practice. Good administration and leadership at each NAC is needed. By gaining an understanding of interviewers’ motivation, and enhancing this, engagement and retention of interviewers may be increased.Electronic supplementary materialThe online version of this article (10.1186/s12909-018-1128-z) contains supplementary material, which is available to authorized users.