Asking applicants what they would do versus what they did do: A meta‐analytic comparison of situational and past behaviour employment interview questions

Abstract

Criterion-related validities and inter-rater reliabilities for structured employment interview studies using situational questions (e.g. ''Assume that you were faced with the following situation . . . what would you do?'') were compared meta-analytically with studies using past behaviour questions (e.g. ''Can you think of a time when . . . what did you do?''). Validities and reliabilities were further analysed in terms of whether descriptively-anchored rating scales were used to judge interviewees' answers, and validities for each question type were also assessed across three levels of job complexity.While both question formats yielded high validity estimates, studies using past behaviour questions, when used with discriptively anchored answer rating scales, yielded a substantially higher mean validity estimate than studies using the situational question format with descriptively-anchored answer rating scales (.63 versus .47). Question type (situational versus past behaviour) was found to moderate interview validity, after controlling for whether studies used answer rating scales. No support was found for the hypothesis that situational questions are less valid for predicting job performance in high-complexity jobs.Sample-weighted mean inter-rater reliabilities were similar for both situational and past behaviour questions, provided that descriptively-anchored rating scales were used (.79 and .77, respectively), although they were slightly lower (.73) for past behaviour question studies lacking such rating scales.

The Impact of Job Complexity and Study Design on Situational and Behavior Description Interview Validity

“…Through our extensive search we were able to find an additional 19 SI studies, giving us a total of 32 SI studies. Taylor and Small's (2002) meta-analysis included analysis of both situational and behavior description interview validity. They had 30 SI studies in their dataset, 23 of which were included 11 in our dataset.…”

“…Through our search we were able to find an additional 9 SI studies, giving us a total of 32 SI studies. Taylor and Small (2002) also had 19 BDI studies, 17 of which were in our dataset. The two studies from their dataset which we did not use were Latham and Skarlicki (1995), where the criterion was organizational citizenship behavior, and Latham and Saari (1984), where the majority of questions were not behavior description in nature but rather appeared to be Ghisellitype (1966) questions such as "Why did you want this job?".…”

The effects of predictor method factors on selection outcomes: A modular approach to personnel selection procedures.

“…For instance, textual and pictorial stimuli might be presented via a paper-and-pencil or computerized medium (PC, tablet, smartphone, etc.). Cognitive load theory, (Sweller, 1988); media richness theory (Daft & Lengel, 1984;Potosky, 2008); multiple resource theory of attention (Wickens, 1984 (Mischel & Shoda, 1995); procedural justice theory (Gilliland, 1993); information processing theory (Hembree, 1992;Kintsch, 1988); cognitive load theory (Sweller, 1988); pragmatic reasoning schema theory (Cheng & Holyoak, 1985) Large effect of no vs. low on validity of personality ratings: meta-analysis (Schaffer & Postlethwaite, 2012) Higher validity for pastbehavior interviews than for situational interviews, controlling for response evaluation consistency (Taylor & Small, 2002) More error variance for no vs. low in personality scales (Robie et al, 2000;Schmit et al, 1995); Less error variance for no vs. high for cognitive tests (Hattrup et al, 1992) Larger cognitive saturation for situational interviews; larger personality saturation for past-behavior interviews: metaanalysis (Berry et al, 2007) No effect of no vs. high on BlackWhite differences in cognitive tests (DeShon et al, 1998;Hattrup et al, 1992) This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.…”

Exploring Methods for Developing Behaviorally Anchored Rating Scales for Evaluating Structured Interview Performance

“…PBQs operate under the assumption that past behavior is one of the best predictors of future behavior (the behavioral consistency principle; Schmitt & Ostroff, 1986;Wernimont & Campbell, 1968). Research has shown that PBQs generally yield higher validity coefficients than SQs (Taylor & Small, 2002). Further, although findings are not entirely consistent, PBQ scores tend to be more strongly related to personality and job experience, whereas SQ scores tend to be more strongly related to cognitive ability or, perhaps, job knowledge (Levashina et al, 2014).…”

“…Standardized, anchored scales to rate responses to each question comprise one element of interview structure (Campion, Palmer, & Campion, 1997). Use of behaviorally anchored rating scales (BARS) tends to increase the reliability and predictive validity of structured interview scores (Taylor & Small, 2002) and may decrease bias against protected groups (Reilly, Bocketti, Maser, & Wennet, 2006). Unfortunately, the amount of time and effort required to create BARS is often excessive (Landy & Farr, 1980;Landy, Farr, Saal, & Freytag, 1976;Maurer, 1997).…”