A Statistical Procedure for Testing Unusually Frequent Exactly Matching Responses and Nearly Matching Responses

Haberman, Shelby J.; Lee, Yi‐Hsuan

doi:10.1002/ets2.12150

Cited by 13 publications

(12 citation statements)

References 27 publications

(44 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast to work on test security by Holland (1996) and Haberman and Lee (2017), the authors do not appear to exploit distractors, despite the availability of an unauthorized answer key with errors. Simply comparing scores based on use of the unauthorized answer key with scores based on the true key can be remarkably informative.…”

Section: Comments On the Specific Papersmentioning

confidence: 71%

Recent Challenges to Maintaining Score Comparability: A Commentary

Dorans

Haberman

2022

J Educational Measurement

Self Cite

View full text Add to dashboard Cite

This special issue addresses a timeless fundamental concern, namely the desire to compare measurements across different instruments, different measurement conditions and different classes of objects being measured. It is timely because, as the overview by Moses and Puhan (2022) notes, much variation in measurement instruments, measurement conditions, and classes of objects is occurring. Some of these changes may be in response to the pandemic, the desire to innovate, or the perceived need to redefine the emphasis of assessment in response to changes in the demand for and acceptability of testing.We address the papers in the following order: Baldwin and Clauser; Puhan and Kim; Jones, Tong, Liu, Borglum and Primoli; Liu and Becker, Moses. Baldwin and Clauser (2022) provide a conceptual framework for thinking about the problem of score comparability in terms of what they call connectives and follow that with examples from the history of innovations in testing that employ different connectives. To address issues of comparability associated with remote proctoring of exams, Puhan and Kim (2022) summarize statistical procedures that can be used to evaluate potential mode effects for both individual items and total scores. Then they use a check on the invariance of linking relationships between the same two forms administered under both standardized conditions at test centers and under remote proctoring conditions in homes of test takers. Jones et al. ( 2022) describe the challenges posed by remote proctoring, and they examine two methods for attempting to detect mode effects that may occur between test center administrations and remote proctoring. The Liu and Becker (2022) paper concerns the effects that remote proctoring has on tests that are administered continuously and that rely on the assumption that item parameter estimates obtained via IRT preequating remain invariant. Moses (2022) discusses existing linking frameworks and their application to increased variation in measurements conditions and to greater use of indirect collateral information to adjust for subpopulation differences. At the extreme, he notes that the invariance of linking functions is simply assumed. We close with some summary comments on the importance of comparing the comparable to fairly assess individuals. Comments on the Specific Papers Baldwin and ClauserBaldwin and Clauser provide a perspective drawn from their experiences with innovations in the field of medical credentialing examinations that is applicable to problems posed by the impact of a medical emergency on credentialing examinations

show abstract

Section: Comments On the Specific Papersmentioning

confidence: 71%

Recent Challenges to Maintaining Score Comparability: A Commentary

Dorans

Haberman

2022

J Educational Measurement

Self Cite

View full text Add to dashboard Cite

show abstract

“…Other limitations include the nonparametric approach in estimating OCCs and response time; that is, we could not use a parametric item response model for item analysis because of the restriction imposed by the large numbers of response selections for the MC.m items—and for the same reason, use of item response time was limited in our study. A modified NRM approach as in Haberman and Lee (2017) could be explored, in which distractor selection and proficiency are conditionally independent given that the test taker's selection was incorrect, even though this approach does not result in the test taker gaining more credit with fewer options. Further studies may also investigate parametric modeling of response selections and response time with appropriate data.…”

Section: Discussionmentioning

confidence: 99%

Using Existing Data to Inform Development of New Item Types

Guo

Ling

Frankel

2020

ETS Research Report Series

View full text Add to dashboard Cite

With advances in technology, researchers and test developers are developing new item types to measure complex skills like problem solving and critical thinking. Analyzing such items is often challenging because of their complicated response patterns, and thus it is important to develop psychometric methods for practitioners and researchers to analyze these new item types. In this study, we describe a generic approach that involves data‐driven analyses and expert feedback from different research areas so that the analysis results can provide valuable information to test developers and researchers on how complex item types contribute to score reliability and validity and on how to make the test more efficient and reliable in measuring complex skills. A real data example was used to illustrate how to identify nonfunctioning options that might be removed from the test and whether partial credit for certain response selections can be considered.

show abstract

“…The answer-copying indices that are most popular include the K-index (e.g., Holland, 1996), the closely related probability of matching incorrect responses (PMIR; e.g., Lewis & Thayer, 1998), the ω index (Wollack, 1997), and the generalized binomial model approach (van der Linden & Sotaridona, 2006). The popular similarity indices include the M4 index (Maynes, 2014) for detecting pairs of examinees one or more of whom may have been involved in test fraud, the index of Wollack and Maynes (2017) that employs cluster-analysis methods (e.g., Everitt, Landau, Leese, & Stahl, 2011) in conjunction with the M4 index (Maynes, 2014) to detect groups of test-takers engaged in collusion, and the analysis for matching of response patterns based on a specialized IRT model (Haberman & Lee, 2017).…”

Section: The Five Types Of Statistical/ Psychometric Approaches To Dementioning

confidence: 99%

Latent-variable Approaches Utilizing Both Item Scores and Response Times To Detect Test Fraud

Sinharay

2021

Open Education Studies

View full text Add to dashboard Cite

There is a growing interest in approaches based on latent-variable models for detecting fraudulent behavior on educational tests. Wollack and Schoenig (2018) noted the presence of five types of statistical/psychometric approaches to detect the three broad types of test fraud that occur in educational tests. This paper includes a brief review of the five types of statistical/psychometric approaches mentioned by Wollack and Schoenig (2018). This paper then includes a more detailed review of the recent approaches for detecting test fraud using both item scores and response times—all of these approaches are based on latent-variable models. A real data example demonstrates the use of two of the approaches.

show abstract

A Statistical Procedure for Testing Unusually Frequent Exactly Matching Responses and Nearly Matching Responses

Cited by 13 publications

References 27 publications

Recent Challenges to Maintaining Score Comparability: A Commentary

Recent Challenges to Maintaining Score Comparability: A Commentary

Using Existing Data to Inform Development of New Item Types

Latent-variable Approaches Utilizing Both Item Scores and Response Times To Detect Test Fraud

Contact Info

Product

Resources

About