Cognitive Task Analysis for Implicit Knowledge About Visual Representations With Similarity Learning Methods

Abstract

Visual representations are prevalent in STEM instruction. To benefit from visuals, students need representational competencies that enable them to see meaningful information. Most research has focused on explicit conceptual representational competencies, but implicit perceptual competencies might also allow students to efficiently see meaningful information in visuals. Most common methods to assess students’ representational competencies rely on verbal explanations or assume explicit attention. However, because perceptual competencies are implicit and not necessarily verbally accessible, these methods are ill‐equipped to assess them. We address these shortcomings with a method that draws on similarity learning, a machine learning technique that detects visual features that account for participants’ responses to triplet comparisons of visuals. In Experiment 1, 614 chemistry students judged the similarity of Lewis structures and in Experiment 2, 489 students judged the similarity of ball‐and‐stick models. Our results showed that our method can detect visual features that drive students’ perception and suggested that students’ conceptual knowledge about molecules informed perceptual competencies through top‐down processes. Furthermore, Experiment 2 tested whether we can improve the efficiency of the method with active sampling. Results showed that random sampling yielded higher accuracy than active sampling for small sample sizes. Together, the experiments provide the first method to assess students’ perceptual competencies implicitly, without requiring verbalization or assuming explicit visual attention. These findings have implications for the design of instructional interventions that help students acquire perceptual representational competencies.

Efficiently Learning Relative Similarity Embeddings with Crowdsourcing

“…Typically, experimentalists require an inordinate number of human responses (about 10,000) to produce an accurate embedding when making a similarity map in 𝑑 = 2 dimensions of 𝑛 = 50 chemistry molecules (Mason et al, 2019). The number of human responses required will scale like 𝒪(𝑛𝑑 log 𝑛), which means that asking about 𝑛 = 100 molecules for 𝑑 = 3 dimensions will likely require about 35,000 responses.…”

Interactive computer assessment and analysis of students’ ability in scientific modeling

“…Conceptual methods focus on analyzing the ''black box'' of cognition (Mason et al, 2019). One of the main techniques is cognitive modeling analysis (Norman, 2008), which creates valuable insights into ''natural mappings'' between cognition and interface.…”

One for All: Simultaneous Metric and Preference Learning over Multiple Users

“…The results in the previous section give guarantees on the generalization error of a learned metric and ideal points when predicting pair responses over X , but do not bound the recovery error of the learned parameters M , { v k } K k=1 with respect to M * and {v * k } K k=1 . Yet, in some settings such as data generated from human responses [22,23] it may be reasonable to assume that a true M * and {v * k } K k=1 do exist that generate the observed data (rather than serving only as a model) and that practitioners may wish to estimate and interpret these latent variables, in which case accurate recovery is critical. Unfortunately, for an arbitrary noise model and loss function, recovering M * and {v * k } K k=1 exactly is generally impossible if the model is not identifiable.…”