Examining the effects of testwiseness in conceptual physics evaluations

Self Cite

134

Research on the test structure of the Force Concept Inventory (FCI) has largely ignored gender, and research on FCI gender effects (often reported as "gender gaps") has seldom interrogated the structure of the test. These rarely crossed streams of research leave open the possibility that the FCI may not be structurally valid across genders, particularly since many reported results come from calculus-based courses where 75% or more of the students are men. We examine the FCI considering both psychometrics and gender disaggregation (while acknowledging this as a binary simplification), and find several problematic questions whose removal decreases the apparent gender gap. We analyze three samples (total N pre ¼ 5391, N post ¼ 5769) looking for gender asymmetries using classical test theory, item response theory, and differential item functioning. The combination of these methods highlights six items that appear substantially unfair to women and two items biased in favor of women. No single physical concept or prior experience unifies these questions, but they are broadly consistent with problematic items identified in previous research. Removing all significantly gender-unfair items halves the gender gap in the main sample in this study. We recommend that instructors using the FCI report the reduced-instrument score as well as the 30-item score, and that credit or other benefits to students not be assigned using the biased items.

Section: Samplementioning

confidence: 99%

Gender fairness within the Force Concept Inventory

Traxler

Henderson

Stewart

et al. 2018

Self Cite

134

“…"Force Graph" items (items [14][15][16][17][18][19][20][21] ask students about the force on a toy car as it moves across a low-friction surface; students select from a number of graphs. "Acceleration Graph" items (items [22][23][24][25][26] ask students to select the graph which correctly represents the acceleration of a toy car moving on a horizontal surface. "Coin Toss -Acceleration" items (items [27][28][29] ask students to select the acceleration of a coin tossed in the air.…”

Section: A the Fmce Instrumentmentioning

confidence: 99%

“…This discussion will focus on the analysis retaining nodes selected by 20% of the students; results retaining nodes selected by 5% and 10% of the students are discussed in RQ3. Modified Module Analysis showed the item groups proposed by Smith and Wittman were being consistently answered using a common misconception: the "Force Sled" (items 1-4, 7), the "Force Graph" (items 14, [16][17][18][19], "Acceleration Graphs" (items [22][23][24][25][26] and "Newton III" (items 30-32, 34, 36, 38) [27]. The "Reversing Direction" subgroup of items (items 8-10, 11-13, 27-29) [27] was not consistently identified as an incorrect answer community.…”

Section: A Research Questionsmentioning

confidence: 99%

Exploring the structure of misconceptions in the Force and Motion Conceptual Evaluation with modified module analysis

Wells

Henderson

Traxler

et al. 2020

Self Cite

Investigating student learning and understanding of conceptual physics is a primary research area within Physics Education Research (PER). Multiple quantitative methods have been employed to analyze commonly used mechanics conceptual inventories: the Force Concept Inventory (FCI) and the Force and Motion Conceptual Evaluation (FMCE). Recently, researchers have applied network analytic techniques to explore the structure of the incorrect responses to the FCI identifying communities of incorrect responses which could be mapped on to common misconceptions. In this study, the method used to analyze the FCI, Modified Module Analysis (MMA), was applied to a large sample of FMCE pretest and post-test responses (Npre = 3956, Npost = 3719). The communities of incorrect responses identified were consistent with the item groups described in previous works. As in the work with the FCI, the network was simplified by only retaining nodes selected by a substantial number of students. Retaining nodes selected by 20% of the students produced communities associated with only four misconceptions. The incorrect response communities identified for men and women were substantially different, as was the change in these communities from pretest to post-test. The 20% threshold was far more restrictive than the 4% threshold applied to the FCI in the prior work which generated similar structures. Retaining nodes selected by 5% or 10% of students generated a large number of complex communities. The communities identified at the 10% threshold were generally associated with common misconceptions producing a far richer set of incorrect communities than the FCI; this may indicate that the FMCE is a superior instrument for characterizing the breadth of student misconceptions about Newtonian mechanics. *

“…Each item has a minimum of six possible responses with some items having nine responses. All items include a "none of the above" response which may cause psychometric problems [44]. Since all items include a none of the above response, each item should suffer from having a distractor which is preferentially not selected by the student; thus no item will be affected more than other items.…”

Section: A the Fmce And The Fcimentioning

confidence: 99%

Multidimensional item response theory and the force and motion conceptual evaluation

Yang

Zabriskie

Stewart

2019

Self Cite

Many studies have examined the structure and properties of the Force Concept Inventory (FCI); however, far less research has investigated the Force and Motion Conceptual Evaluation (FMCE). This study applied Multidimensional Item Response Theory (MIRT) to a sample of N ¼ 4528 FMCE post-test responses. Exploratory factor analysis showed that 5, 9, and 10-factor models optimized some fit statistics. The FMCE uses extensive blocking of items into groups with a common stem; these blocks factored together in most models. A confirmatory analysis, which constrained the MIRT models to a theoretical model constructed from expert solutions, produced a model requiring only 8 principles, fundamental reasoning steps. This was substantially fewer than the 19 principles identified in the FCI by a previous study. Correlation analysis also demonstrated that the two instruments were very dissimilar. The reduced number of principles and the repetition of items using a single principle allowed the extraction of eight single-principle subscales, seven with Cronbach's alpha greater than the 0.7 required for acceptable internal consistency. The differences between the FCI and the FMCE suggest that the two instruments could provide complementary, but different, information about student understanding of Newton's laws with the FCI measuring an integrated Newtonian force concept and the FMCE measuring details of that force concept.