Assessing Student Learning and Improving Instruction with Transition Matrices

Abstract

For common multiple-choice assessments, we can investigate progress in student understanding by creating simple transition matrices that identify the percentage of students who select each possible pre-/post-test answer combination on each question of a diagnostic exam such as the Force Concept Inventory. In order to create a transition matrix, we first rank answer choices from worst to best using item response curves. This allows us to determine changes in understanding of concepts and misconceptions even when students do not get the correct answer on the post-test. Our work outlines the utility of transition matrices and the availability of a new tool for adoption by high school and college level physics teachers who use a common assessment. We then construct transition probabilities matrices and find that students who chose an incorrect answer to a question on the pre-test are equally likely to choose the correct answer on the post-test regardless of which wrong answer she or he selected on the pre-test.

Quantitatively ranking incorrect responses to multiple-choice questions using item response theory

“…This assumption is based on the premise that students who understand more about Newtonian physics are more likely to choose better incorrect answers than students who understand less physics, and these students are also more likely to choose a greater number of correct responses. This assumption is consistent with previous work that has used item response curves (IRCs) to examine and rank incorrect responses on both the FCI and the FMCE [15][16][17][18]. We expand on this prior work by using a nested-logit item response theory (IRT) model to simultaneously estimate students' overall understanding of Newtonian mechanics (the IRT latent trait, or person parameter) and determine how closely each response choice correlates with a high level of understanding using the estimated parameters of the model [19][20][21][22][23][24].…”

Showing the dynamics of student thinking as measured by the FMCE

“…Given the strong correlation between the student parameter and the total score on the FCI [4], Morris et al use the total score as the independent variable rather than estimations of a latent trait. Walter and Morris expanded on this work by using IRCs to rank incorrect responses on the FCI from more to less sophisticated [8]. Their main claim is that an incorrect response that is more popular among higher-scoring students represents a higher level of understanding than a response that is most popular among lower-scoring students.…”

“…We determined students' total scores out of 33 possible points based on typical scoring recommendations for the first 43 questions of the FMCE [13]. Unfortunately, we were unable to use the method described by Walter and Morris to rank incorrect responses to FMCE questions [8]. The inclusion of up to four additional answer choices (nine on some FMCE questions compared to the FCI's five) made it impossible to distinguish between many of the options.…”

“…Rankings of incorrect responses allow us to examine students' answers on the pre-and post-tests to gauge how much learning occurred during the semester. Walter and Morris suggest using transition matrices that show the percentage of students who gave each pre/post answer pair for a particular question [8]. Tables II and III show the transition matrices from two different data sets (from different schools) for questions 2 and 14, respectively.…”

“…Unfortunately, all incorrect answers are treated equally in typical analyses, and little to no attention is paid to why a student would select a particular incorrect answer. These concerns have been addressed by various researchers: Smith, Wittmann, and Carter interpreted the most common incorrect response to each FMCE question through a resources perspective and used Model Analysis to show shifts in modes of student thinking [5,6], Thornton identified several "student views" based on interpretations of various incorrect responses on a small subset of FMCE questions [7], and more recently Walter and Morris have ranked incorrect responses to FCI questions (from most to least sophisticated) using Item Response Curves (IRCs) and shown how students progress through answer choices using transition matrices [8].…”