Does the choice of response time threshold procedure substantially affect inferences concerning the identification and exclusion of rapid guessing responses? A meta-analysis

Abstract

Background In testing contexts that are predominately concerned with power, rapid guessing (RG) has the potential to undermine the validity of inferences made from educational assessments, as such responses are unreflective of the knowledge, skills, and abilities assessed. Given this concern, practitioners/researchers have utilized a multitude of response time threshold procedures that classify RG responses in these contexts based on either the use of no empirical data (e.g., an arbitrary time limit), response time distributions, and the combination of response time and accuracy information. As there is little understanding of how these procedures compare to each other, this meta-analysis sought to investigate whether threshold typology is related to differences in descriptive, measurement property, and performance outcomes in these contexts. Methods Studies were sampled that: (a) employed two or more response time (RT) threshold procedures to identify and exclude RG responses on the same computer-administered low-stakes power test; and (b) evaluated differences between procedures on the proportion of RG responses and responders, measurement properties, and test performance. Results Based on as many as 86 effect sizes, our findings indicated non-negligible differences between RT threshold procedures in the proportion of RG responses and responders. The largest differences for these outcomes were observed between procedures using no empirical data and those relying on response time and accuracy information. However, these differences were not related to variability in aggregate-level measurement properties and test performance. Conclusions When filtering RG responses to improve inferences concerning item properties and group score outcomes, the actual threshold procedure chosen may be of less importance than the act of identifying such deleterious responses. However, given the conservative nature of RT thresholds that use no empirical data, practitioners may look to avoid the use of these procedures when making inferences at the individual-level, given their potential for underclassifying RG.

To What Degree Does Rapid Guessing Distort Aggregated Test Scores? A Meta-analytic Investigation

“…For instance, if primary studies employed response time thresholds that were too strict, RG may have been underidentified (Wise & Kuhfeld, 2020), leading to potentially understated score distortion. This may be reflected in our descriptive results in which large differences in the percentage of RG responses identified by response time threshold procedures were observed, supporting prior research (Rios & Deng, 2021). Although researchers have utilized multiple proxies to identify RG (e.g., Harmes & Wise, 2016;Sahin & Colvin, 2020), they are all indirect indicators of examinee behaviors that make strong assumptions to infer the occurrence of RG behavior.…”

“…Response times have been the most popular source of information utilized to identify RG in both the literature (e.g., Rios et al, 2017) and operational testing contexts (e.g., Wise & Kuhfeld, 2020). In this approach, a RT threshold is defined in which any response provided in less time is considered to be RG (for details, see Rios & Deng, 2021;Wise, 2017). The extensive adoption of RT as a proxy for RG is associated with its ability to be unobtrusive (i.e., examinees are unaware that their behavior is tracked) and identify RG on an item-by-item basis for each examinee (e.g., Silm et al, 2020).…”

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

“…In this case, the latent ignorability assumption is unlikely to hold, and scaling models that rely on it (see [ 4 , 12 ]) will result in biased and unfair country comparisons. We are skeptical that the decision of whether a missing item response is scored as wrong should be based on a particular response time threshold [ 166 , 172 , 173 ]. Students can also be simply instructed to quickly skip items that they are not probably able to solve.…”

How Tenable are Modeling Assumptions Around Rapid Guessing Behavior? Results From a Large Corpus of Low-Stakes Assessments

“…Within the response time threshold framework, it is presumed that response times monotonically increase based on examinee effort, and thus, response time distributions between RG and solution behavior (SB) are anticipated to be distinct (i.e., bimodal). Given this assumption, identification of RG requires that a threshold is identified to differentiate between these distributions, which can be accomplished by using heuristic rules (e.g., any response provided is less than three seconds is a rapid guess), inspecting response time distributions, and combining response time and accuracy information (see Rios & Deng, 2021;Soland et al, 2019).…”