Test equity in developing short version conceptual inventories: A case study on the Conceptual Survey of Electricity and Magnetism

Abstract

Standardized concept inventories (CIs) have been widely used in science, technology, engineering, and mathematics education for assessment of student learning. In practice, there have been concerns regarding the length of the test and possible test-retest memory effect. To address these issues, a recent study developed a method to split a CI into two equivalent short CIs, which have been shown to provide equivalent mean score measures. However, the previous approach does not fully examine common requirements of test equating. This study extends the existing method with test equating analysis to form a revised algorithm for developing and validating equivalent short CIs. The method is applied to split the Conceptual Survey of Electricity and Magnetism (CSEM) into two half-length CSEMs (HCSEMs). Through a series of test-equating and validation analysis, the HCSEMs are confirmed to measure the same content clusters and the same construct of students' understanding as that of the original CSEM at a similar level of reliability regarding their question design features. Furthermore, a best performing equating function to convert scores among the two HCSEMs and the CSEM is identified through comparisons of four commonly used equating functions. The conversions are further validated using the existing data sets as well as a data set collected from an empirical study using randomized testing. The results confirm that the three CIs can provide equivalent measures in different subpopulations. Overall, the results from this study have shown that the revised algorithm can provide a more complete framework in developing and validating equivalent short CIs. In addition, this method can also be applied to develop and validate parallel CIs in general.

Linking and comparing short and full-length concept inventories of electricity and magnetism using item response theory

“…In a number of recent studies, Han et al and Xiao et al have developed and improved a set of methods for creating and validating equivalent short CIs based on the original full-length CIs [5,6]. In Han et al's study, the algorithm for developing equivalent and short parallel CIs involves five steps: (i) classify the items from the original CI into categories based on content analysis; (ii) split the items into two short versions to minimize their differences in both content and difficulty; (iii) compare the assessment features of the short CIs and the original CI; (iv) establish linear regression models between the total scores of any two of the CIs; and (v) reexamine the score conversion model using a different sample [6].…”

“…Recently, Xiao et al further extended Han et al's algorithm with an emphasis on incorporating standard practices in test equating, which are expected to meet five common requirements including equal construct, equal reliability, symmetry, equity, and population invariance requirements [5,[8][9][10]. In the new study, the linear regression model of equating used in Han et al [6] was replaced with other methods due to the asymmetry between the regression relations obtained from two equatable tests [4,11].…”

“…In the new study, the linear regression model of equating used in Han et al [6] was replaced with other methods due to the asymmetry between the regression relations obtained from two equatable tests [4,11]. In Xiao et al's study, a symmetric equating model was used and a seven-step revised algorithm for developing and validating short CIs was proposed [5]: (i) conduct content analysis on the CI questions to develop splitting categories, (ii) create and adjust the question assignment in short CIs based on existing data and content categories, (iii) examine the equal construct requirement, (iv) examine the equal reliability requirement, (v) establish equating functions among different CIs, (vi) examine the equity and population invariance requirements, and (vii) apply new CIs in randomized testing to validate the CIs and equating model. Typically, the development of short CIs would go through multiple cycles of revisions and reevaluation until all requirements are met.…”

“…Typically, the development of short CIs would go through multiple cycles of revisions and reevaluation until all requirements are met. Following the revised algorithm, two short versions of CSEM have been developed and validated [5].…”

“…In this study, the item response theory (IRT) framework was incorporated to extend an existing algorithm for developing short equivalent CIs [5]. The extended algorithm was successfully applied to develop and validate two short equivalent versions of BEMA.…”