Using cluster analysis to identify patterns in students’ responses to contextually different conceptual problems

Abstract

This study examined the evolution of student responses to seven contextually different versions of two Force Concept Inventory questions in an introductory physics course at the University of Arkansas. The consistency in answering the closely related questions evolved little over the seven-question exam. A model for the state of student knowledge involving the probability of selecting one of the multiple-choice answers was developed. Criteria for using clustering algorithms to extract model parameters were explored and it was found that the overlap between the probability distributions of the model vectors was an important parameter in characterizing the cluster models. The course data were then clustered and the extracted model showed that students largely fit into two groups both pre-and postinstruction: one that answered all questions correctly with high probability and one that selected the distracter representing the same misconception with high probability. For the course studied, 14% of the students were left with persistent misconceptions post instruction on a static force problem and 30% on a dynamic Newton's third law problem. These students selected the answer representing the predominant misconception slightly more consistently postinstruction, indicating that the course studied had been ineffective at moving this subgroup of students nearer a Newtonian force concept and had instead moved them slightly farther away from a correct conceptual understanding of these two problems. The consistency in answering pairs of problems with varied physical contexts is shown to be an important supplementary statistic to the score on the problems and suggests that the inclusion of such problem pairs in future conceptual inventories would be efficacious. Multiple, contextually varied questions further probe the structure of students' knowledge. To allow working instructors to make use of the additional insight gained from cluster analysis, it is our hope that the physics education research community will make these methods available though their Web sites.

Synthesis of discipline-based education research in physics

“…Other publications explore these and other analysis methods in greater detail, including item response theory [121,419,420], cluster analysis [421,422], Rasch model based analysis [423], concentration analysis [424], and model analysis [399].…”

Multidimensional item response theory and the Force Concept Inventory

“…Fitting a model implementing the structure suggested in the original FCI paper on the set of items 1,2,3,4,7,8,12,13,14,15,17,19,20,21,25,28 …”

“…The structure of student reasoning on the FCI has also been investigated by methods such as model analysis that require the input of a partial model of the concepts measured by the FCI [19]. Model analysis was later shown to be exact only in certain limiting cases [12]. For a summary of these exploratory and nonexploratory methods, see the review by Ding and Beichner [20].…”

“…These have included purely exploratory or descriptive methods such as factor analysis [7][8][9], module analysis [10], cluster analysis [11,12], item response theory [13][14][15][16], and item response curves [17,18]. The structure of student reasoning on the FCI has also been investigated by methods such as model analysis that require the input of a partial model of the concepts measured by the FCI [19].…”

A quantitative analysis of Educational Data through the Comparison between Hierarchical and Not-Hierarchical Clustering

“…In a recent paper, Stewart et al (2012) analyze the student responses to contextually different versions of Force Concept Inventory questions, by using a model analysis for the state of student knowledge and ClA methods to characterize the distribution of students' answers. The authors conclude that ClA is an effective method to extract the underlying subgroups in student data and that additional insight may be gained from a further analysis of clustering results.…”