Applying Item Response Theory Methods to Examine the Impact of Different Response Formats

Abstract

In aptitude and achievement tests, different response formats are usually used. A fundamental distinction must be made between the class of multiple-choice formats and the constructed response formats. Previous studies have examined the impact of different response formats applying traditional statistical approaches, but these influences can also be studied using methods of item response theory to deal with incomplete data. Response formats can influence item attributes in two ways: different response formats could cause items to measure different latent traits or they could contribute differently to item difficulty. In contrast to previous research, the present study examines the impact of response formats on item attributes of a language awareness test applying different item response theory models. Results indicate that although the language awareness test contains items with different response formats, only one latent trait is measured; no format-specific dimensions were found. Response formats do, however, have a distinct impact on the difficulty of the items. In addition to the effects of the three administered item types, a fourth component that makes items more difficult was identified.

Assessing writing ability in primary education : on the evaluation of text quality and text complexity

“…As Table 13 shows, no significant differences were found. Similar to earlier findings by Hohensinn and Kubinger (2011) and In'nami and Koizumi (2009), the CR items seemed slightly more difficult.…”

“…Given this correlation, both the CR and the MC formats appear to measure the same latent trait to a great extent, but the formats may elicit different skills and/or assess skills differently (cf. Hohensinn & Kubinger, 2011;Lissitz & Hou, 2012).…”

Dealing With Item Nonresponse in Large‐Scale Cognitive Assessments: The Impact of Missing Data Methods on Estimated Explanatory Relationships

“…Current practices in low-stakes educational large-scale achievement tests involve treating unplanned missing values as incorrect or fractionally correct responses or ignoring them in the scaling (see, e.g., PISA, Adams & Wu, 2002;TIMSS [Third International Mathematics and Science Study], Martin, Gregory, & Stemler, 2000; NAEP [National Assessment of Educational Progress], Allen, Donoghue, & Schoeps, 2001;NEPS [National Educational Panel Study], Pohl & Carstensen, 2012). Research on these types of missing data approaches showed bias on item and person parameter estimates when missing values were scored as incorrect (Culbertson, 2011;De Ayala, Plake, & Impara, 2001;Finch, 2008;Hohensinn & Kubinger, 2011;Holman & Glas, 2005;Pohl, Gräfe, & Rose, 2014;Rose et al, 2010). The method of fractionally correct scoring performed slightly better but also resulted in bias, especially when missing values were MNAR (De Ayala et al, 2001;Finch, 2008).…”

Modeling Local Item Dependence Due to Common Test Format With a Multidimensional Rasch Model

“…The Rasch testlet model yields itself very well to modeling LID due to its common method. Instead of forming group factors based on common passages we build them based on common test formats (Hohensinn & Kubinger, 2011). That is, each item is forced to load on a target ability dimension and its pertinent method dimension.…”