To What Degree Does Rapid Guessing Distort Aggregated Test Scores? A Meta-analytic Investigation

Rios, Joseph A.; Deng, Jiayi; Ihlenfeldt, Samuel D.

doi:10.1080/10627197.2022.2110465

Cited by 7 publications

(5 citation statements)

References 51 publications

(62 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, data were generated for a 50-item test. Although this test length falls within the range commonly seen in low-stakes testing contexts (Rios et al, 2022), it is likely that due to time constraints, shorter tests may be present in applied settings. The results of this study may not generalize to other test lengths as the accuracy of item and ability parameter estimates will largely be dependent on possessing adequate statistical power for the estimated models.…”

Section: Discussionmentioning

confidence: 99%

“…Although unmotivated simulees could vary in their number of RG responses, the total percentage of RG responses across all disengaged simulees was manipulated to be equal to either 10% or 20% of all item responses. The former level is reflective of the average percentage of RG responses detected in a recent meta-analysis of low-stakes assessments (Rios et al, 2022), while the latter is an upper-end percentage seen in previously applied analyses (e.g., Goldhammer et al, 2016). Upon identifying the number of RG responses for each unmotivated simulee, RG was imputed by replacing the true response probability with the chance level (.25) for the items with the highest RG propensity probabilities; all remaining items were treated as true SB in the RG classification matrix.…”

Section: Methodsmentioning

confidence: 97%

“…Second, to reflect that RG is largely reflective of random responding (Wise & DeMars, 2006), RG was simulated in the data by substituting a proportion of effortful probabilities with chance likelihoods (.25; assuming four response options per item). This process was completed by manipulating RG responses in the data matrix for a disengaged subsample of simulees (hereon referred to as the RG or unmotivated subsample), which comprised 28% of the total sample—an average observed across numerous low-stakes assessment contexts (Rios et al, 2022). Upon creating an item response matrix, the third step consisted of producing a RG classification matrix in which errors were imputed prior to parameter estimation.…”

Section: Methodsmentioning

confidence: 99%

See 2 more Smart Citations

A Comparison of Response Time Threshold Scoring Procedures in Mitigating Bias From Rapid Guessing Behavior

Rios

Deng

2023

Educational and Psychological Measurement

Self Cite

View full text Add to dashboard Cite

Rapid guessing (RG) is a form of non-effortful responding that is characterized by short response latencies. This construct-irrelevant behavior has been shown in previous research to bias inferences concerning measurement properties and scores. To mitigate these deleterious effects, a number of response time threshold scoring procedures have been proposed, which recode RG responses (e.g., treat them as incorrect or missing, or impute probable values) and then estimate parameters for the recoded dataset using a unidimensional or multidimensional IRT model. To date, there have been limited attempts to compare these methods under the possibility that RG may be misclassified in practice. To address this shortcoming, the present simulation study compared item and ability parameter recovery for four scoring procedures by manipulating sample size, the linear relationship between RG propensity and ability, the percentage of RG responses, and the type and rate of RG misclassifications. Results demonstrated two general trends. First, across all conditions, treating RG responses as incorrect produced the largest degree of combined systematic and random error (larger than ignoring RG). Second, the remaining scoring approaches generally provided equal accuracy in parameter recovery when RG was perfectly identified; however, the multidimensional IRT approach was susceptible to increased error as misclassification rates grew. Overall, the findings suggest that recoding RG as missing and employing a unidimensional IRT model is a promising approach.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 97%

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

A Comparison of Response Time Threshold Scoring Procedures in Mitigating Bias From Rapid Guessing Behavior

Rios

Deng

2023

Educational and Psychological Measurement

Self Cite

View full text Add to dashboard Cite

show abstract

“…Numerous studies demonstrated that disengaged responding is a serious concern in educational large-scale assessments, with estimated prevalence rates up to 28%, depending on various aspects such as the content domain, setting, assessment mode, test-taker characteristics, country, or detection method ( Goldhammer et al, 2017 ; Kroehne et al, 2020 ; Nagy & Ulitzsch, 2022 ; Rios et al, 2022 ). Disengaged responding manifests itself on the item-by-person level with engagement rates varying both between examinees and items ( Ulitzsch et al, 2020 ).…”

Section: Types Of Test-taking Behaviormentioning

confidence: 99%

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators

Welling

Gnambs

Carstensen

2023

Educational and Psychological Measurement

View full text Add to dashboard Cite

Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In this study, three process data variables—text reread, item revisit, and answer change—were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed.

show abstract

“…For instance, RG has been shown to bias item parameter and person (ability) estimates (Rios et al, 2017), biases equating and scaling analyses , and distorts psychometric item properties (e.g., item information; van Barnevald, 2007). Importantly, RG has been found to be a prevalent behavior in LSAs, with a recent a meta-analysis reporting that up to 28% of examinees engage in RG to some degree at least once throughout the course of a testing event (Rios, Deng, & Ihlenfeldt, 2022). These findings highlight the importance of identifying RG and correcting its adverse effects; however, the impact of RG on model fit and factor-analytic reliability (MF&R) are not well understood.…”

mentioning

confidence: 99%

The Impact of Rapid Guessing on Model Fit and Factor-Analytic Reliability: An Exploratory Analysis

Martinez¹,

Rios²

2023

Preprint

View full text Add to dashboard Cite

Rapid guessing (RG) is form of non-effortful responding whereby examinees provide a response in a timeframe that is incommensurate with the amount of time needed to thoroughly engage with the item. Previous research has found that RG leads to biased parameter estimates, biased ability estimates, and measurement non-invariance if not properly accounted for. The consequences of RG on model fit and factor-analytic reliability (MF&R), however, are not well understood. To address this gap in the literature, the present study explores the effects of rapid guessing on model fit and reliability across a corpus of 20 diverse low-stakes assessments. Three RG scoring approaches (Naïve, Penalized, Effort-moderated with imputation) were compared across four model fit indices (CFI, TLI, RMSEA, SRMR) and two reliability coefficients (McDonald’s omega and Cronbach’s alpha). We found evidence that model fit is influenced by choice of scoring procedure, with the effort-moderated with imputation method producing better model fit than naïve and penalized scoring procedures when RG rates were greater. RG was also found to differentially impact reliability indices, however no systematic trends across scoring method and RG rates emerged. Implications of the research, recommendations for practitioners, and a discussion of future directions are discussed.

show abstract

To What Degree Does Rapid Guessing Distort Aggregated Test Scores? A Meta-analytic Investigation

Cited by 7 publications

References 51 publications

A Comparison of Response Time Threshold Scoring Procedures in Mitigating Bias From Rapid Guessing Behavior

A Comparison of Response Time Threshold Scoring Procedures in Mitigating Bias From Rapid Guessing Behavior

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators

The Impact of Rapid Guessing on Model Fit and Factor-Analytic Reliability: An Exploratory Analysis

Contact Info

Product

Resources

About