Validating two questions in the Force Concept Inventory with subquestions

Abstract

In this study, we evaluate the structural validity of Q.16 and Q.7 in the Force Concept Inventory (FCI). We address whether respondents who answer Q.16 and Q.7 correctly actually have an understanding of the concepts of physics tested in the questions. To examine respondents' levels of understanding, we use subquestions that test them on concepts believed to be required to answer the actual FCI questions. Our sample size comprises 111 respondents; we derive false-positive ratios for prelearners and postlearners and then statistically test the difference between them. We find a difference at the 0.05 significance level for both Q.16 and Q.7, implying that it is possible for postlearners to answer both questions without an understanding of the concepts of physics tested in the questions; therefore, the structures of Q.16 and Q.7 are invalid. In this study, we only evaluate the validity of these two FCI questions; we do not assess the validity of previous studies that have compared total FCI scores.

Generating a growth-oriented partial credit grading model for the Force Concept Inventory

“…These values can be found in Table II. An interesting result from the presented analysis was its ability to detect potentially malfunctioning questions [38]. Questions 11,14,15,16, and 29 were all found to be potentially malfunctioning, but upon investigation only questions 16 and 29 were determined as likely malfunctioning, which is in agreement with previous research results [4,12,28].…”

“…Because of this slight modification in the situation, option E is now awarded a partial credit score of 0.947. This question has been identified in the past as malfunctioning [4,28,29]. It has been observed that students are able to answer this question correctly through the use of Newton's first law, not through Newton's third [4].…”

“…It is suggested that question 29 be examined in more depth by following the procedure used in Ref. [28], and by conducting student interviews or giving free response questionnaires for this question. Until a study of this nature is done, question 29 can only be concluded to be potentially malfunctioning for currently unknown reasons.…”

Investigating students’ seriousness during selected conceptual inventory surveys

“…Yasuda et al show that while question 5 scores on the FCI are marginally affected by erroneous reasoning, questions 6, 7, and 16 are more prone to guessing. These questions return a high percentage of false positives as students seem to reach the right answer while using erroneous conceptual reasoning [38,39].…”

Towards quantification of the FCI’s validity: the effect of false positives

“…Since the two forces act on different bodies, this is an incorrect application of Newton's First Law that is frequently seen with students taking introductory physics. This "false positive"-obtaining the correct answer despite not correctly understanding the content tested by the question-on Q.16 was observed by the FCI creators as well as other researchers validating the FCI [3][4][5]. False positives are a source of systematic error, inflating student scores on the FCI.…”