Generative Response Modeling: Leveraging the Computer as a Test Delivery Medium

Abstract

Generative response modeling is an approach to test development and response modeling that calls for the creation of items in such a way that the parameters of the items on some response model can be anticipated through knowledge of the psychological processes and knowledge required to respond to the item. That is, the computer would not merely retrieve an item from a database, as is the case in adaptive testing, but would compose it, or assist in doing so, according to desired specifications. This approach to assessment has implications for both the economics and validity of computer administered tests. To illustrate the concept, a system for measuring writing skills will be outlined where the examinee is expected to rewrite sentences, rather than just recognize errors in a sentence, using a multiple choice format. The possibility of estimating the psychometric parameters of items based on a psychological analysis of the response process will then be examined and shown to be feasible. Such estimates are less precise than estimates based on large samples of test takers. A Monte Carlo study is presented to investigate the possibility of compensating for that imprecision when estimating ability or proficiency. The paper concludes that a generative approach is feasible, and can be a mechanism for taking advantage of the considerable investment required for computer-based testing.Generative response modeling 3 Generative response modeling: Leveraging the computer as a test delivery medium Although the use of computers in testing was envisioned several decades ago (e.g., Green, 1964;Weiss, 1978), it is only recently that nationally administered tests have become operationally feasible for licensing exams (e.g., Lewis and Sheehan, 1990), placement (e.g., Ward, 1984), and admission tests (e.g., Wainer, H., Dorans, N. J., Flaugher, R., Green, B. F., Mislevy, R. J., Steinberg, L., Thissen, D., 1990). Interestingly, the major hurdles along the way to implementation were not psychometric per se, but rather organizational, economical, and technological. Indeed, from the point of view of a testing organization, the implementation of computer-based testing requires a substantial "reengineering" of major aspects of the test production process. However, had a research foundation for creative use ofthe computer not been laid out well before the wide availability of computing power (e.g., Lord, 1980), it is likely that today's computer-based tests would not have taken full advantage of the computer as a medium of delivery. Research must continue to insure that this will be the case in the future and that better tests, not just more economical or convenient ones, will be developed. One possibility is to take advantage of the computer to administer performance-based or open-ended exams that may not have been feasible in a paper and pencil format. A major hurdle in the implementation of performance-based exams is the cost of scoring. Thus, the use of computers to score openended responses (e.g., Bejar, 1991;Bennett, 1993;Bra...

A Feasibility Study of On‐the‐fly Item Generation in Adaptive Testing

“…As can be seen, no bias is observed under any condition. Thus, as has been observed elsewhere (Bejar, 1996;Embretson, 1999), the impact of lack of isomorphicism is primarily in measurement precision, although the losses at some levels of ability appear to be minimal. This outcome is fortunate, as a loss of precision can be compensated, but bias would be more difficult to correct.…”

“…In this report, we have adopted the term item modeling to refer this approach. An item model (Bejar, 1996) can be thought of as a procedure for instantiating isomorphic items -items that contain comparable content and are exchangeable psychometrically. We view item modeling as construct-driven because it entails an understanding of the goals of the assessment and the application of pertinent psychological research.…”

Information-based methods for evaluating the semantics of automatically generated test items

“…The outcome of this algorithmic transcription is often referred to as item modeling. [3][4][5][6] The item modeling process requires the identification of elements within the assessment task so that these elements can be used to create large set of items. [1,7] These generated items may or may not be similar to one another and thus the lexical similarity among the generated items is often unknown.…”

Calibration of Polytomous Item Families Using Bayesian Hierarchical Modeling

“…While efforts to create new items are laborious for multiple-choice (MC) items, the same for complex constructed-response (CR) tasks are even more challenging. Due to the effort, expense, and occasionally inconsistent item quality associated with traditional item production, there is an increasing interest in using item models (Bejar, 1996) to guide production of items, automatically or manually, with similar conceptual and statistical properties. Items produced from a single item model, whether by automatic item generation (AIG) systems (Irvine & Kyllonen, 2002) or by rigorous manual procedures, are related to one another through the common generating model, and therefore constitute a family of related items.…”