Scoring Concept Maps: An Expert Map‐Based Scheme Weighted for Relationships

Abstract

The use of student‐constructed concept maps in assessment is congruent with the changing emphases set forth by the National Science Education Standards. Authorities have expressed concern about concept map scoring systems and their associated validity and reliability. They favor methods that employ expert/criterion maps as referents and emphasize the use of accurate concept relationships in deriving scores, which have been found to correlate with performance on standardized tests. In this study, student constructed concept maps (n= 17) that emerged from post‐instructional interviews about chlorofluorocarbons were scored against a teacher‐expert map using a scheme weighted for relationships. Interrater reliability for the scoring scheme was high (r= .959). Students' map scores correlated highly with their scores on the California Achievement Test component total (r= .729) and moderately with their Pathfinder index (r= .474), the latter believed to be an excellent measure of structural knowledge. A revised map score, derived only from relationships containing one or more of the concepts employed in Pathfinder analysis, was a statistically significant (p= .031) predictor of the Pathfinder index. The findings of this study support the recommendations of others to use expert referents and emphasize concept relationships in assessing concept maps.

Making Sense of Knowledge Integration Maps

“…Concept maps can reveal students' knowledge organization by showing connections, clusters of concepts, hierarchical levels, and cross-links between concepts from different levels . Concept map analysis, especially of more constrained forms, has been found to be reliable and valid (Markham et al, 1994;Michael, 1995;Ruiz-Primo et al, 1997, 2001Rye & Rubba, 2002;Shavelson et al, 2005;Stoddart et al, 2000;Yin et al, 2005). Less constrained forms of concept maps can include many different kinds of concepts and connections.…”

“…This chapter suggests that scoring propositions using a knowledge integration rubric can reveal a greater variety of students' alternative concepts than a direct comparison to an expert-generated benchmark map (for examples of direct comparisons see Chang, Sung, & Chen, 2001;Cline, Brewster, & Fell, 2009;Herl, O'Neil, Chung, Dennis, & Lee, 1997;Rye & Rubba, 2002). The knowledge integration concept map rubric acknowledges different ways concepts can be expressed.…”

“…Propositions, the composite of two concepts, a link label, and an arrow can be evaluated in order to learn about students' understanding. It can be decided to evaluate all propositions equally, to weight certain propositions more than others (Rye & Rubba, 2002), or to analyze only certain indicator propositions (Ruiz-Primo et al, 2009). Yin et al (2005) showed that scoring each individual proposition on a four-point individual proposition scale, summed up to a "total accuracy score," provided the best validity: 0 for scientifically wrong or irrelevant propositions, 1 for partially incorrect propositions, 2 for correct but scientifically "thin" propositions, and 3 for scientifically correct and strong propositions.…”

Assessing Teacher’s Integration of Technology and Literacy in Elementary Science Classrooms in Kuwait

“…Some have used down scales to score the quality of concepts and schemes in the maps (Rye &Rubba, 2002) or whether participants correctly identified concepts vs. sub-concepts (Bayram, 1995) within the map structure itself. Another scoring system was developed to compare experts in learning based on the comparison of concepts maps between a novice and expert (Ruiz-Primo, Shavelson, & Schultz, 2001).…”

Element matching in concept maps

“…Meaningful learning processes exhibited in the organization of student concept maps include: (1) new concept learning in the node names, (2) subsumption in specific items connected to more general previous knowledge, (3) progressive differentiation in multiple hierarchical layers, and (4) integrative reconciliation in cross-links connecting separate branches of a hierarchical tree. At least 128 different methods of creating and scoring CMs have been proposed [9] but existing computerized CM-systems do not include algorithms to support student assessment. A system that counts levels of hierarchy, matches corresponding hierarchical clusters, and automatically identifies cross-links would be useful for concept map evaluation.…”