Transition Matrices: A Tool to Assess Student Learning and Improve Instruction

Morris, Gary A.; Walter, Paul; Skees, Spencer; Schwartz, Samantha

doi:10.1119/1.4976661

Cited by 6 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These values can be found in Table II. An interesting result from the presented analysis was its ability to detect potentially malfunctioning questions [38]. Questions 11,14,15,16, and 29 were all found to be potentially malfunctioning, but upon investigation only questions 16 and 29 were determined as likely malfunctioning, which is in agreement with previous research results [4,12,28].…”

Section: Limitations and Future Worksupporting

confidence: 88%

“…So, if a distractor has a partial credit score above 0.75, then it may be an indication that the associated question is not being interpreted properly or is malfunctioning in some other manner. Looking over the partial credit scores in Table II shows that questions 11,14,15,16, and 29 all have distractors with scores greater than 0.75. It should be noted that this method is not a "catch all" and that some items which pass this investigation may still be malfunctioning.…”

Section: B Potentially Malfunctioning Questionsmentioning

confidence: 99%

See 1 more Smart Citation

Generating a growth-oriented partial credit grading model for the Force Concept Inventory

Eaton

Johnson

Willoughby

2019

Phys. Rev. Phys. Educ. Res.

View full text Add to dashboard Cite

Traditionally, multiple choice assessments are graded in a dichotomous manner, where selecting the correct option for a question awards 1 point and the selection of an incorrect option awards 0 points. As a result of this grading scheme, all incorrect response options are treated as being equally incorrect regardless of potential differences in their relative correctness, intentional or otherwise. We propose a partial credit grading model for the Force Concept Inventory (FCI) that allots points to the incorrect responses. This was done using slope parameters from the two-parameter logistics nominal response model (2PLNRM), a model from polytomous item response theory (PIRT). The resulting scores from the partial credit model represent student growth towards a proper Newtonian mindset, as measured by the FCI. Observations indicate that this model accounts for student progression through prominent misconceptions (i.e., impetus) as their world views become more Newtonian. As expected, we find that student total scores increase as a result of the model, but the average overall gains on the assessment are essentially unchanged. The data used in this analysis were maintained and organized by PhysPort and included about 20 000 responses from first semester introductory physics courses at multiple universities. Ultimately, this partial credit model allows instructors to more accurately gauge the growth of their students over the course of instruction. Additionally, as a result of these partial credit scores, we are able to identify potentially malfunctioning questions on the FCI that may be sources of error in measuring student abilities.

show abstract

Section: Limitations and Future Worksupporting

confidence: 88%

Section: B Potentially Malfunctioning Questionsmentioning

confidence: 99%

Generating a growth-oriented partial credit grading model for the Force Concept Inventory

Eaton

Johnson

Willoughby

2019

Phys. Rev. Phys. Educ. Res.

View full text Add to dashboard Cite

show abstract

“…1 and 2, the tool also provides feedback on changes in understanding of concepts and remaining misconceptions that are listed in Hestenes et al (1992) (using the 1995 revised Tables I and II) [1]. The FCI Transition Matrices Tool is openly available for physics instructors on the American Modeling Teachers Association website [4].…”

Section: Fci Transition Matrices Toolmentioning

confidence: 99%

Assessing Student Learning and Improving Instruction with Transition Matrices

Walter¹,

Morris²

2016

2016 Physics Education Research Conference Proceedings

Self Cite

View full text Add to dashboard Cite

For common multiple-choice assessments, we can investigate progress in student understanding by creating simple transition matrices that identify the percentage of students who select each possible pre-/post-test answer combination on each question of a diagnostic exam such as the Force Concept Inventory. In order to create a transition matrix, we first rank answer choices from worst to best using item response curves. This allows us to determine changes in understanding of concepts and misconceptions even when students do not get the correct answer on the post-test. Our work outlines the utility of transition matrices and the availability of a new tool for adoption by high school and college level physics teachers who use a common assessment. We then construct transition probabilities matrices and find that students who chose an incorrect answer to a question on the pre-test are equally likely to choose the correct answer on the post-test regardless of which wrong answer she or he selected on the pre-test.

show abstract

“…In recent years, many researchers have explored methods for reporting student growth using methods that are more informative than comparisons of single-number scores generated from dichotomously scored items. Morris et al analyze pre-/post-instruction concept inventory data to incorporate all of the answer choices on an item to construct transition matrices to observe changes in student understanding [4]. In another example, Pérez-Lemonche et al apply multidimensional item response theory to study the effect of answer choices that highlight particular student misconceptions and act as distractors [5].…”

Section: Introductionmentioning

confidence: 99%

Comparing pre/post item response curves to identify changes in misconceptions

Walter¹,

Smith²

2021

2021 Physics Education Research Conference Proceedings

View full text Add to dashboard Cite

We use quantitative measures and visual inspection to compare the item response curves (IRCs) of matched pre-/post-instruction Force Concept Inventory (FCI) data. We find that the IRCs are not static; the pre-instruction IRCs differ from the post-instruction IRCs by more than can be explained by random chance. This result is also the case for a subpopulation consisting of students who make little or no gains on the FCI, suggesting that learning is taking place even when scores do not change appreciably. We consider three items where students make substantial progress (item 4) or little progress (items 14 and 21) compared to overall changes in FCI scores.

show abstract

Transition Matrices: A Tool to Assess Student Learning and Improve Instruction

Cited by 6 publications

References 12 publications

Generating a growth-oriented partial credit grading model for the Force Concept Inventory

Generating a growth-oriented partial credit grading model for the Force Concept Inventory

Assessing Student Learning and Improving Instruction with Transition Matrices

Comparing pre/post item response curves to identify changes in misconceptions

Contact Info

Product

Resources

About