Classical test theory and item response theory comparison of the brief electricity and magnetism assessment and the conceptual survey of electricity and magnetism

Abstract

For proper assessment selection understanding the statistical similarities amongst assessments that measure the same, or very similar, topics is imperative. This study seeks to extend the comparative analysis between the brief electricity and magnetism assessment (BEMA) and the conceptual survey of electricity and magnetism (CSEM) presented by Pollock. This is accomplished by using large samples (N BEMA ¼ 5368 and N CSEM ¼ 9905) within classical test theory (CTT) and item response theory (IRT) frameworks. For the IRT comparison, after consideration of the conceptual content addressed in each assessment, it was assumed that each of these assessments are measuring the same student latent ability (θ), specifically a student's ability to do introductory electricity and magnetism. Via a CTT and IRT analysis it was found that both assessments are essentially equal in overall difficulty. Classical item analysis applied to 7 questions used by both assessments revealed that each assessment functions slightly differently internally. The test information curves found from IRT show that the CSEM has superior information compared to the BEMA in estimating student latent abilities for the entire range of typical latent abilities achieved by students on each assessment, θ ≈ −2 to θ ≈ 3. Information in this case is interpreted as how well a student's latent ability was estimated by an assessment as a function of latent ability. When the circuits questions are removed from the BEMA the majority of the information is lost in the θ ≈ 0 to θ ≈ 2 range. This means the circuits questions on the BEMA are information heavy for higher ability scores. So, special considerations should be made as to which assessment a study uses depending on the specific questions a researcher is attempting to answer.

Linking and comparing short and full-length concept inventories of electricity and magnetism using item response theory

“…It was suggested that the CSEM and BEMA were nearly equivalent in measuring students' understanding of static electricity and magnetism at the introductory level. Recently, Eaton et al corroborated Pollock's findings [18]. The two CIs were compared using both the CTT and IRT approaches.…”

“…Furthermore, although the overall difficulty of CSEM and BEMA has been compared as being equivalent in previous studies [17,18], there is a gap in the literature on finer grained comparisons of individual students' performances on these two CIs which is worth exploring. In addition, this study will produce two short CIs of BEMA, which make a total collection of 6 CIs including short and long versions on electricity and magnetism.…”

“…To answer the first research question, the full-length BEMA test will be divided into two short CIs following the method in Xiao et al [5], but with an extension into using the IRT framework in addition to the CTT approaches for test evaluation and validation. To address the issue of nonidentical common items in Eaton et al's study [18], the study will use identical items in HBEMAs and HCSEMs as common items for linking between the two full-length tests. Using identical common linking items, more stable linking results between the full-length BEMA and CESM can be obtained and used to develop a score conversion model between the two commonly used CIs.…”

Como Contribuir No Ensino Do Eletromagnetismo a Partir Das Dificuldades Conceituais Dos Alunos?

“…Conforme Eaton et al (2019), que fez aplicações utilizando o BEMA e CSEM, diz que a confiabilidade do teste refere-se à consistência que este teste tem em dar uma nota específica para o respondente; ressalta que valores abaixo de 0,8 fazem com que o teste conceitual esteja apto a relacionar grupos, mas não indivíduos. O Alfa de Cronbach é igual a:…”

Identifying a preinstruction to postinstruction factor model for the Force Concept Inventory within a multitrait item response theory framework

“…Many studies have tested these assessments in an attempt to understand the psychometric properties of these tools, see Refs. [2][3][4][5][6][7][8][9][10][11].…”