An evaluation of scanpath-comparison and machine-learning classification algorithms used to study the dynamics of analogy making

Abstract

In recent years, eyetracking has begun to be used to study the dynamics of analogy making. Numerous scanpathcomparison algorithms and machine-learning techniques are available that can be applied to the raw eyetracking data. We show how scanpath-comparison algorithms, combined with multidimensional scaling and a classification algorithm, can be used to resolve an outstanding question in analogy making-namely, whether or not children's and adults' strategies in solving analogy problems are different. (They are.) We show which of these scanpath-comparison algorithms is best suited to the kinds of analogy problems that have formed the basis of much analogy-making research over the years. Furthermore, we use machine-learning classification algorithms to examine the item-to-item saccade vectors making up these scanpaths. We show which of these algorithms best predicts, from very early on in a trial, on the basis of the frequency of various item-to-item saccades, whether a child or an adult is doing the problem. This type of analysis can also be used to predict, on the basis of the item-to-item saccade dynamics in the first third of a trial, whether or not a problem will be solved correctly.Keywords Eyetracking algorithms . Jarodzka algorithm . LDA . SVM . Analogy strategies Traditionally, analogy making has been studied statically. Participants typically see a pair of related images (the Bbase pair^), along with a third image and a number of candidate target images. One of these target images-the Bcorrect analogical match^-is supposed to be related to the third image in the same way that the base items were related to one another. The participant's task is to identify the correct analogical match. Correct and incorrect answers (and, sometimes, reaction times) are recorded and analyzed. However, these studies could not capture-and in fairness, were not designed to capture-the dynamic aspects of solving an analogy problem. As such, they shed essentially no light on the question of what strategies were adopted during the course of solving analogy problems.In this article, we introduce a novel means of studying the dynamic aspects of analogy making in both children and adults. The proposed methodology involves combining eyetracking, multidimensional scaling (MDS), and neuralnetwork classification algorithms, as well as using machinelearning algorithms to analyze the component vectors making up participants' scanpaths. In what follows, we will briefly describe each of these techniques and show how they can be combined successfully in the context of analogy making.Although the purpose of this article is, first and foremost, a methodological one, it is important to note that the development of these techniques has allowed us (French & Thibaut, 2014;Thibaut & French, 2016;Thibaut, French, Missault, Gérard, & Glady, 2011) to answer, for what we believe to be the first time, a long-standing question in the field of analogymaking-namely, do children and adults use the same (or very similar) search-space strategies when...

Understanding the What and When of Analogical Reasoning Across Analogy Formats: An Eye‐Tracking and Machine Learning Approach

“…Indeed, most studies do not analyze the temporal dynamics of trials. In preliminary studies that later led to Thibaut and French (2016) and French et al (2017), we started with a finer five-slice analysis which gave overly complex results (interactions) essentially similar to the ones reported here. Here, a three-slice approach allows us to separate early explorations of the semantic space from late explorations in the trial which can be interpreted as decisional.…”

“…This confirms the previous (behavioral) analysis and suggests that, for complex problems, which require the space of possible solutions to be explored more thoroughly, participants look at all potential solutions, including the unrelated distractors throughout the course of the problem. Even though we do not provide the behavioral analyses, we ran an identical SVM+LOOCV analysis to see which strategies led to correct or incorrect answers in the two groups of problems, to parallel the ones provided by Thibaut and French (2016) and French et al (2017). Indeed, finding that these two types of conditions had different profiles in adults would be interesting, as French et al (2017) focused on children only.…”

“…We used a simple and powerful classification technique, a support vector machine (SVM, Vapnik, 1999, combined with a leave-one-out cross-validation, LOOCV, see below). Using this technique, in French et al (2017), we were able to predict whether an adult or a child was solving a particular analogy problem and, this, in the first third of the trial, based only on the distribution of the transitions (saccades). Using an identical approach in the present paper, we considered how well various transition profiles predicted the difficulty of a problem under scrutiny (see the Results section for more details).…”

multimatch-gaze: The MultiMatch algorithm for gaze path comparison in Python

“…It reveals commonalities and differences of viewing behavior within and between observers, and is used to study how people explore visual information. For example, scan path comparisons are used to study analogy-making (French, Glady, & Thibaut, 2017), visual exploration and imagery (Johansson, Holsanova, & Holmqvist, 2006), habituation in repetitive visual search (Burmester & Mast, 2010), or spatial attention allocation in dynamic scenes (Mital, Smith, Hill, & Henderson, 2011). The method is applied within individuals as a measure of change (Burmester & Mast, 2010), or across samples to study group differences (French et al, 2017).…”

Peak correlation classifier (PCC) applied to FTIR spectra: a novel means of identifying toxic substances in mixtures

“…We have tested the SVM in a number of other contexts [25, 26] and found it to be clearly superior in its classification performance to ANNs and LDA. However, one of the key factors in the use of the SVM is that it is more robust than the other standard classifiers tested.…”