Influence of response instructions and response format on applicant perceptions of a situational judgement test for medical school selection

Abstract

BackgroundThis study examined the influence of two Situational Judgement Test (SJT) design features (response instructions and response format) on applicant perceptions. Additionally, we investigated demographic subgroup differences in applicant perceptions of an SJT.MethodsMedical school applicants (N = 372) responded to an online survey on applicant perceptions, including a description and two example items of an SJT. Respondents randomly received one of four SJT versions (should do-rating, should do-pick-one, would do-rating, would do-pick-one). They rated overall favourability and items on four procedural justice factors (face validity, applicant differentiation, study relatedness and chance to perform) and ease-of-cheating. Additionally, applicant perceptions were compared for subgroups based on gender, ethnic background and first-generation university status.ResultsApplicants rated would-do instructions as easier to cheat than should-do instructions. Rating formats received more favourable judgements than pick-one formats on applicant differentiation, study-relatedness, chance to perform and ease of cheating. No significant main effect for demographic subgroup on applicant perceptions was found, but significant interaction effects showed that certain subgroups might have more pronounced preferences for certain SJT design features. Specifically, ethnic minority applicants – but not ethnic majority applicants – showed greater preference for should-do than would-do instructions. Additionally, first-generation university students – but not non-first-generation university students – were more favourable of rating formats than of pick-one formats.ConclusionsFindings indicate that changing SJT design features may positively affect applicant perceptions by promoting procedural justice factors and reducing perceived ease of cheating and that response instructions and response format can increase the attractiveness of SJTs for minority applicants.Electronic supplementary materialThe online version of this article (10.1186/s12909-018-1390-0) contains supplementary material, which is available to authorized users.

Development and Implementation of Situational Judgment Tests as an Evaluation Method for Training Oncology Physicians: Application in the KOKON-KTO Training

“…As presented in the study, SJTs might be useful not just as a selection method (e.g., display of performance) but also as an assessment tool for evaluations (e.g., for training curricula). In future studies, the presentation of the developed SJTs may be varied to detect possible adherence factors such as design features [ 32 , 33 ].…”

Use of situational judgment tests for assessing non‐cognitive attributes of final year dental students

“…Following this were three main parts: (i) demographic information: name, age, gender, study year, name of institution; (ii) SJT of 70 items, 10 items per domain; and (iii) a prevalidated feedback form consisting of 10 questions on a five-point Likert scale (1: strongly disagree to 5: strongly agree) about perceived predicative validity, perceived fairness, face validity, applicant differentiation, study relatedness, chance to perform, ease of cheating, relevance, perceived content validity, and item difficulty. 30 All questions were forced marked for participants to avoid missing data, and option "only allow survey to be taken once" was marked to avoid ballot box stuffing. 31 Only Ayesha Fahim had access to data, which was deidentified prior to analysis.…”

Applicant perceptions of selection methods for health professions education: Rationales and subgroup differences

“…Previous research suggests that prospective students with lower socio‐economic and ethnic minority backgrounds more often demonstrate negative perceptions of selection and are less confident in their ability to be successfully admitted 3,11 . These subgroups of applicants are shown to have poorer selection outcomes in general, 12,13 which may be partially explained by poorer motivation due to more negative perceptions 14 …”

“…To our knowledge, only one single‐site study was executed in the field of HPE. This research focused on SJTs and concluded that ethnic minority students and first‐generation university students had other preferences in SJT format compared with their traditional counterparts 14 . Another single‐site study that was conducted at an undergraduate psychology programme included a wide range of selection methods but only focused on gender as a background variable 2 .…”

“…3,11 These subgroups of applicants are shown to have poorer selection outcomes in general, 12,13 which may be partially explained by poorer motivation due to more negative perceptions. 14 Applicant perceptions of selection are commonly described using organisational justice theory, 15 which distinguishes between distributive and procedural justice. Distributive justice describes the perceived fairness of the distribution of the outcome of the selection procedures, while procedural justice refers to the perceived fairness of the selection procedure and methods that are used to generate this outcome.…”