Test Length and Decision Quality in Personnel Selection: When Is Short Too Short?

Abstract

Personnel selection shows an enduring need for short stand-alone tests consisting of, say, 5 to 15 items. Despite their efficiency, short tests are more vulnerable to measurement error than longer test versions. Consequently, the question arises to what extent reducing test length deteriorates decision quality due to increased impact of measurement error. A distinction was made between decision quality at the group level and the individual level. Using simulations, we found that short tests had a large negative impact on individual-level decision quality, in particular for selecting suited candidates when base rates or selection ratios are low. Negative effects on group-level decision quality were smaller than those for individual-level decision quality. Results were similar for dichotomous-item tests and rating-scale tests, and also for top-down and cut-score selection.Personnel selection shows the increasing tendency to use short tests consisting of, say, 5 to 15 items, for making decisions about individuals applying for a job (e.g., 322KRUYEN, EMONS, SIJTSMA have been aware of the influence of test length on test-score reliability under idealized conditions of parallel test parts, which are often individual items but in fact any test parts (like subsets of items) that are parallel. The message of the Spearman-Brown prophecy formula is that reliability increases as the test grows longer but the reverse message is that reliability decreases as the test grows shorter. Although somewhat weaker, these trends are also effective if test parts are not parallel, which is common in real tests.Given this century-old knowledge, the trend of using short tests is intriguing, and the question how the use of short and, hence, less-reliable test scores affects the quality of decision-making thus becomes paramount. Emons, Sijtsma, and Meijer (2007) investigated this question for cut-score selection in a computational study and concluded that as test length decreases, the certainty by which one makes the right decisions about individuals can be impaired and become worrisome. Here, we extended their study to include interesting results at the group level in addition to the individual level. Group level is important because hiring many unsuited applicants causes high organizational costs. Individual level needs to be considered because incorrectly selected applicants may gain bad work experiences, while incorrectly rejected applicants may decide to sue the organization for being rejected on the basis of inadequate testing procedures. Assessing group-level and individuallevel decision quality requires different perspectives, as we point out later. We studied the influence of test shortening on decision quality for groups and individuals in five selection scenarios that are regularly discussed in the literature on personnel selection. Our study sheds light on the issue whether short tests can be used responsibly in personnel selection and if so, under which conditions this is possible. BACKGROUND

On the Shortcomings of Shortened Tests: A Literature Review

“…Loss of measurement precision means that decisions about individuals are more uncertain to a degree that uncertainty may become unacceptable for many decisions. The result may be a large number of decision errors (Emons et al, 2007;Kruyen et al, 2012).…”

Assessing Individual Change Using Short Tests and Questionnaires

“…First, Emons et al (2007) and Kruyen et al (2012) consistently found minor effects of number of response categories on individual decision making. Therefore, the results based on items with five response categories can be generalized to items having other numbers of categories.…”

Some Relationships Between Cronbach’s Alpha and the Spearman-Brown Formula

“…using Crobach's alpha), it is frequently desirable to predict the reliability of a test score of a test that measures the same construct but has a different length. For example, in recent years there is an increasing interest in short or shortened tests (Kruyen, Emons, and Sijtsma 2012). A recent literature review is presented in Kruyen, Emons, and Sijtsma (2013).…”