An item response curves analysis of the Force Concept Inventory

Abstract

Several years ago, we introduced the idea of item response curves (IRC), a simplistic form of item response theory (IRT), to the physics education research community as a way to examine item performance on diagnostic instruments such as the Force Concept Inventory (FCI). We noted that a full-blown analysis using IRT would be a next logical step, which several authors have since taken. In this paper, we show that our simple approach not only yields similar conclusions in the analysis of the performance of items on the FCI to the more sophisticated and complex IRT analyses but also permits additional insights by characterizing both the correct and incorrect answer choices. Our IRC approach can be applied to a variety of multiple-choice assessments but, as applied to a carefully designed instrument such as the FCI, allows us to probe student understanding as a function of ability level through an examination of each answer choice. We imagine that physics teachers could use IRC analysis to identify prominent misconceptions and tailor their instruction to combat those misconceptions, fulfilling the FCI authors’ original intentions for its use. Furthermore, the IRC analysis can assist test designers to improve their assessments by identifying nonfunctioning distractors that can be replaced with distractors attractive to students at various ability levels.

Validating two questions in the Force Concept Inventory with subquestions

“…Similarly, if we evaluate and unify the systematic error from the validation of the distractor and the context [8][9][10][11], we could derive the systematic error of the FCI, thereby allowing us to compare the results considering these inadequacies without modifying them and to effectively measure the effect of physics education. This conclusion would enable our study to provide a positive and far-reaching effect in the field of physics education research.…”

“…For future work, we plan to carry out the qualitative analysis of actual answer choices and patterns as well as quantitative analysis such as item response curves [9,10]. Moreover, we plan to disambiguate our data in order to assess the false-positive ratios by major area and analyze whether they differ among these populations.…”

“…Hestenes and colleagues evaluated the validity of the wording and diagrams in its questions [1,7], while Rebello and Zollman analyzed the validity of the distractors in the questions by comparing students' responses to four FCI open-ended questions [8]. Morris and colleagues also evaluated the validity of the distractors by analyzing the item response curves [9,10], and Stewart and colleagues validated the contexts of the questions using a ten-question context-modified test [11].…”

Gender fairness within the Force Concept Inventory

“…Five of the items had difficulty parameters outside of the desired range (items 1, 6, and 12 with P > 0.8 and items 17 and 26 with P < 0.2), with none having discrimination less than 0.2. Morris et al reported the item averages of 4500 students pooling data from multiple institutions and reported FCI items 5, 17, and 26 with P < 0.2, but no items with P > 0.8 [5]. Osborn Popp, Meltzer, and Megowan-Romanowicz reported FCI item level scores for 4775 high school students.…”

“…The Force Concept Inventory (FCI) [1] has been studied using tools such as factor analysis [2,3], item response theory [4,5], and network analysis [6]. Though these investigations have probed the structure and validity of the test, they have primarily treated student data as a single undifferentiated sample and have not studied gender effects.…”