Two New Models for Item Preknowledge

Abstract

To evaluate preknowledge detection methods, researchers often conduct simulation studies in which they use models to generate the data. In this article, we propose two new models to represent item preknowledge. Contrary to existing models, we allow the impact of preknowledge to vary across persons and items in order to better represent situations that are encountered in practice. We use three real data sets to evaluate the fit of the new models with respect to two types of preknowledge: items only, and items and the correct answer key. Results show that the two new models provide the best fit compared to several other existing preknowledge models. Furthermore, model parameter estimates were found to vary substantially depending on the type of preknowledge being considered, indicating that answer key disclosure has a profound impact on testing behavior.

Using Item Scores and Distractors to Detect Item Compromise and Preknowledge

“…Thus, EWP responding to extremely difficult items were expected to select the (incorrect) keyed response with a probability of 0.9, whereas EWP responding to extremely easy items were expected to select the (incorrect) keyed response with a probability of 0.5. This probability was conditioned on item easiness rather than examinee ability because previous research has shown that when the answer key is disclosed, items tend to represent the dominant source of variability (Gorney & Wollack, 2022b).…”

Using item scores and response times in person‐fit assessment

“…preknowledge exponent preknowledge slope models of Gorney and Wollack (2022b). The probability of answering a contaminated item correctly was modified such that P * (X i = 1| ) = P(X i = 1| ) 0.1 , and the mean of the log RT distribution was modified such that M * i ( ) = M i ( ) × 0.…”

“…After simulating the uncontaminated data (using the procedure of Study 1), the item score and RT distributions were modified to reflect aberrant behaviour. To simulate preknowledge, we applied the preknowledge exponent and preknowledge slope models of Gorney and Wollack (2022b). The probability of answering a contaminated item correctly was modified such that , and the mean of the log RT distribution was modified such that .…”