A Comparison of the Results of Many-Facet Rasch Analyses Based on Crossed and Judge Pair Designs

Abstract

Citation: İlhan, M. (2016). A comparison of the results of many-facet Rasch analyses based on crossed and judge pair designs. AbstractThe aim of this study was to compare the results of many-facet Rasch analyses based on crossed and judge pair designs. The study was conducted with 168 eighth grade students and five judges. The study data were collected using an achievement test with open-ended questions and a holistic rubric that was used to rate the responses. In the data collection process, the achievement test was initially administered to students. The students' responses were rated by five judges; the ratings provided the data set for the crossed design. In the judge pair design, the students' responses to the test items were rated by any two of the five judges. Then the researcher conducted the many-facet Rasch analysis and compared the analysis outputs of the two designs. It was determined that the difference between the elements of the facets included in the analysis was demonstrated more effectively in the crossed design than the judge pair design. In the judge pair design, however, the study made significant distinctions between the elements of the facets. Moreover, the study found that the relative agreement between the ability estimations calculated based on crossed and judge pair designs was very high, and there was no significant difference between the ability estimations reported by the two designs. Therefore, it is preferable to choose judge pair design for examinations with a large number of participants as it will be less costly.

Modification and validation of the mixed‐format Engineering Concept Assessment for middle school students using many‐facet Rasch measurement

“…Implementing a crossed or mixed design can be time consuming, costly, and result in rater fatigue when rating a large number of students' responses. Also, research conducted by Ilhan (2016) demonstrated that resultant student measures were highly consistent across the two design types applied to the same data set consisting of 168 eighth-grade students, five judges, and ten CR items applying the many-faceted Rasch model. It is recommended to adopt the judge-pair design for when judging a large number of responses such as in the current study (Boone et al, 2016).…”