Audiovisual Learning in Dyslexic and Typical Adults: Modulating Influences of Location and Context Consistency

Abstract

Learning to read involves efficient binding of visual to auditory information. Aberrant cross-modal binding skill has been observed in both children and adults with developmental dyslexia. Here, we examine the contribution of episodic memory to acquisition of novel cross-modal bindings in typical and dyslexic adult readers. Participants gradually learned arbitrary associations between unfamiliar Mandarin Chinese characters and English-like pseudowords over multiple exposures, simulating the early stages of letter-to-letter sound mapping. The novel cross-modal bindings were presented in consistent or varied locations (i.e., screen positions), and within consistent or varied contexts (i.e., co-occurring distractor items). Our goal was to examine the contribution, if any, of these episodic memory cues (i.e., the contextual and spatial properties of the stimuli) to binding acquisition, and investigate the extent to which readers with and without dyslexia would differ in their reliance on episodic memory during the learning process. Participants were tested on their ability to recognize and recall the bindings both during training and then in post-training tasks. We tracked participants’ eye movements remotely with their personal webcams to assess whether they would re-fixate relevant empty screen locations upon hearing an auditory cue—indicative of episodic memory retrieval—and the extent to which the so-called “looking-at-nothing behavior” would modulate recognition of the novel bindings. Readers with dyslexia both recognized and recalled significantly fewer bindings than typical readers, providing further evidence of their persistent difficulties with cross-modal binding. Looking-at-nothing behavior was generally associated with higher recognition error rates for both groups, a pattern that was particularly more evident in later blocks for bindings encoded in the inconsistent location condition. Our findings also show that whilst readers with and without dyslexia are capable of using stimulus consistencies in the input—both location and context—to assist in audiovisual learning, readers with dyslexia appear particularly reliant on consistent contextual information. Taken together, our results suggest that whilst readers with dyslexia fail to efficiently learn audiovisual binding as a function of stimulus frequency, they are able to use stimulus consistency—aided by episodic recall—to assist in the learning process.

A Scoping Review of Webcam Eye Tracking in Learning and Education

“…Gaze data were recorded with Webgazer.js and processed in proportion to each AOI (AOI corresponds to a paragraph on the stimuli). Calabrich et al (2021b) used Webgazer.js operating at a sampling rate of 60 Hz (depending on the screen refresh rate) to investigate audiovisual learning in dyslexic adult readers and intact adults. An algorithm to detect fixations from the raw data was used for the analysis.…”

“…In the case of a specific reading disorder, learning how to read and the reading itself can be a very challenging task. Calabrich et al (2021aCalabrich et al ( , 2021b explored cross-modal bindings and episodic memory in dyslexic and intact adult readers. The reader's attention is also closely related to working with the text and reading.…”