Complex Composites: Issues That Arise in Combining Different Modes of Assessment

Abstract

Data from the California Learning Assessment System are used to examine certain characteristics of tests designed as the composites of items of different modes. The characteristics include rater severity, test information, and definition of the latent variable. Three different assessment modes-multiple-choice, open-ended, and investigation items (the latter two are referred to as performance-based modes)-were combined in a test across three different test forms. Rater severity was investigated by incorporating a rater parameter for each rater in an item response model that then was used to analyze the data. Some rater severities were found to be quite extreme, and the impact of this variation in rater severities on both total scores and trait level estimates was examined. Within-rater variation in rater severity also was examined and was found to have significant variation. The information contribution of the three modes was compared. Performance-based items provided more information than multiple-choice items and also provided greatest precision for higher levels of the latent variable. A projection-like method was applied to investigate the effects of assessment mode on the definition of the latent variable. The multiple-choice items added information to the performance-based variable. The results of the analysis also showed that the projection-like method did not practically differ from the results when the latent trait was defined jointly by both the multiple-choice and the performance-based items. Index terms: equating, linking, multiple assessment modes, polytomous item response models, rater effects. Multiple-choice (MC) items have been used widely in psychological and educational testing for many years. Administrative convenience and computerized scoring make them very convenient. However, MC items have been criticized as being inadequate to fully assess examinees' abilities. Moreover, test-wiseness may seriously contaminate the measurement. Recently, there has been an increased interest in performance-based (PB) items (or constructed-response items) as an alternative to Mac items. A PB item refers to any item format that requires the examinee to generate a response in any way other than selecting from a short list of alternative answers as in MC items (Pollack, Rock, & Jenkins, 1992). The different types of response formats, such as Mac items and the many types of PB items, are referred to here as different assessment modes. The main advantages of PB items are that: (1) they provide a more direct representation of content specifications (face validity and content validity), (2) they provide more diagnostic information about examinees' learning difficulties from their responses, (3) examinees prefer them to Mac items, and (4) the test formats may stimulate the teaching of important skills, such as problem solving and essay writing (Grima & Liang, 1992). However, Mac items are more economical to score and have well-established patterns of reliability. PB items are more difficult to score objectively and re...

Real‐Time Feedback on Rater Drift in Constructed‐Response Items: An Example From the Golden State Examination

“…In line with earlier research on the statewide tests administered by the CDE (Wilson & Case, 2000;Wilson & Wang, 1995) and research by others from as early as the 1930s (e.g., Ashbum, 1938;Eells, 1930) significant variability between raters was found in their essay scoring. Moreover, significant variation was also found within raters overqtime, confirming findings reported by Braun (1988).…”

“…An advantage of IRT also is that it can easily handle data that are obtained with a scoring scale with a limited range. Several authors have used IRT models to document the impact of rater effects on student scores (e.g., Engelhard, 1994Engelhard, , 1996Lunz, Wright, & Linacre, 1990;Myford & Mislevy, 1995;Patz, 1996;Wilson & Wang, 1995;Wolfe & Myford, 1997).…”

Applications and Extensions of MCMC in IRT: Multiple Item Types, Missing Data, and Rated Responses

“…This uncertainty, if not closely monitored, can have devastating consequences. Wilson and Wang (1995) have described the observed impact of varying rater severities on the scores of students in CLAS mathematics performance tasks: they found that the expected raw score differences resulting from scorer severity differences could be as large as 2.0 score points (on a 6-point scale), and that these raw score effects would result in a difference of 10 percentile points for typical students on the particular test under study. Koretz, Stecher, Klein, and McCaffrey (1994) show that Vermont's portfolio assessment program was unable to report school-level scores due primarily to unacceptably high variability between raters.…”

“…Software to apply re- stricted cases of the LLTM (so-called facets models) has been developed by Linacre (1989), as has software that can estimate models specified under the full LLTM approach (Adams & Wilson, 1996;Adams, Wilson, & Wu, 1997;Ponocny & Ponocny-Seliger, 1997). The technique has been applied to rater effect estimation by Engelhard (1994Engelhard ( , 1996, Myford and Mislevy (1995), and Wilson and Wang (1995). The LLTM rater model for a dichotomous item j taken by examinee /' and rated by rater r, has IRF PUT = W* = 1M,P,) = 1+ ex P -' ( e,-P;-p r r (1?)…”

Using Statistical-Based Rating Sheets to Measure Oral Test Inter-Rater Reliability

“…Although recognizing advantages to performance-based examinations (i.e., simulations and performance observations) versus multiple choice examinations, Wilson & Wang (1995, pp.52-53) discuss concerns for rater effects in performance-based examinations, including inter-rater variation in rater severity and within-rater variation of rater severity. Are some raters harder or easier than others?…”