Using ScanMatch scores to understand differences in eye movements between correct and incorrect solvers on physics problems

Abstract

Using a ScanMatch algorithm we investigate scan path differences between subjects who answer physics problems correctly and incorrectly. This algorithm bins a saccade sequence spatially and temporally, recodes this information to create a sequence of letters representing fixation location, duration and order, and compares two sequences to generate a similarity score. We recorded eye movements of 24 individuals on six physics problems containing diagrams with areas consistent with a novice-like response and areas of high perceptual salience. We calculated average ScanMatch similarity scores comparing correct solvers to one another (C-C), incorrect solvers to one another (I-I), and correct solvers to incorrect solvers (C-I). We found statistically significant differences between the C-C and I-I comparisons on only one of the problems. This seems to imply that top down processes relying on incorrect domain knowledge, rather than bottom up processes driven by perceptual salience, determine the eye movements of incorrect solvers. IntroductionResearchers have found consistent patterns of wrong answers to many simple conceptual physics questions [Trowbridge and McDermott 1980;McDermott et. al 1987]. Several cognitive topdown explanations have been provided, including misconceptions formed through interactions with the natural world or misapplication of conceptual resources [Docktor and Mestre 2011]. However, recent claims by Heckler [2011] have suggested a perceptual basis for students' incorrect answers, which are based on attention being directed to the most perceptually salient and plausibly relevant features in a problem. The most salient features capture attention through perceptual processes and less salient features have little opportunity to be considered. Heckler shows some evidence for perceptually-driven responses; however, no eye movement data supporting this hypothesis is provided. Further, he does not provide a specific definition of salience. Therefore, incorrect answers may be governed either by top-down processes relying on incorrectly learned or applied information, or by bottom-up perceptual processes resulting in certain elements capturing attention and leading to activation of reasoning resources based on theseelements.An eye-movement study was used to test these competing hypotheses. Introductory and graduate physics students answered conceptual physics problems regarding a diagram [Madsen et al., 2011]. Three areas of interest (AOIs) were defined for each diagram. First, thematically-relevant AOIs that contained information necessary to correctly answer the question were determined by experts in physics. Second, novice-like AOIs were defined based on coded interview data from novices [Madsen et al. 2011], and third, perceptually salient AOIs were defined as the area(s) on the diagram with the highest saliency rating according to the salience maps produced by a computational algorithm [Itti 2000]. For each problem, the percentage of time spent in each type of interest area was compared be...

Cultural differences in attention: Eye movement evidence from a comparative visual search task

“…This form of analysis was introduced by Miellet et al (2010) and can be used to explore two issues. First, if the exploration strategies are reliably different between the two cultural groups, then the intra-group scanpath comparisons of Saudi participants and of British participants should yield higher scores than an inter-group scanpath comparison that includes all participants (see Madsen et al, 2012;Miellet et al 2010). Second, one might predict that an individualistic style would be more likely to yield higher ScanMatch scores, since a systematic approach to search (i.e.…”

“…These comparisons were carried out for every stimulus, and then averaged across conditions. This approach of arranging ScanMatch data was previously used by Miellet et al (2010) and Madsen et al (2012), and it provides a means of comparing the scores of different groups with each other, adding a greater interpretive value to the scores than when comparing them with absolute scale. If the cultural groups tended to search the stimuli differently, then we would expect the inter-group (S-B) scores to be lower than the intra-group (S-S, B-B) scores.…”

Does It MultiMatch? What Scanpath Comparison Tells us About Task Performance in Teams

“…ScanMatch is one notable example (Cristino et al, 2010). It has been used to compare the scanpaths of physics problem solvers (Madsen et al, 2012), discover the preferences of individuals with autism (Król & Król, 2020), and study complex visual search patterns (Frame et al, 2019). This method is based on the Needleman-Wunsch algorithm (Needleman & Wunsch, 1970) that was created to compare biological sequences.…”

Detection of Depression and Its High-Risk Group using Scanpath Comparison Based on Semantic Information

“…In this decade, several methods for scanpath comparison have been successfully applied in cognitive studies of visual information processing, such as scene perception [8], reading [9], and visual searching [10]. Scanpath comparison can be used to understand differences in eye movements between correct and incorrect solvers on physics problems [11]. Also, Novices and experts could be distinguished by the comparison of their scanpath [12] [13].…”