The Reliability and Validity of Procedural Memory Assessments Used in Second Language Acquisition Research

Abstract

Evidence for the role of procedural memory in second language (L2) acquisition has emerged in our field. However, little is known about the reliability and validity of the procedural memory measures used in this research. The present study (N = 119) examined the reliability and the convergent and discriminant validity of three assessments that have previously been used to examine procedural memory learning ability in L2 acquisition, the dual-task Weather Prediction Task (DT-WPT), the Alternating Serial Reaction Time Task (ASRT), and the Tower of London (TOL). Measures of declarative memory learning ability were also collected. For reliability, the DT-WPT and TOL tasks met acceptable standards. For validity, an exploratory factor analysis did not provide evidence for convergent validity, but the ASRT and the TOL showed reasonable discriminant validity with declarative memory measures. We argue that the ASRT may provide the purest engagement of procedural memory learning ability, although more reliable dependent measures for this task should be considered. The Serial Reaction Time task also appears promising, although we recommend further consideration of this task as the present analyses were post hoc and based on a smaller sample. We discuss these results regarding the assessment of procedural memory learning ability as well as implications for implicit language aptitude.

The complexity of measuring reliability in learning tasks: An illustration using the Alternating Serial Reaction Time task

“…Since the majority of studies focus on learning performance that can be assessed in a single session, assessing internal consistency is preferred over test-retest reliability. Buffington et al (2021) also administered multiple experimental tasks to a group of 99 subjects, including the ASRT task, and calculated split-half reliability. The Spearman-Brown split-half reliability of ASRT was found to be only a moderate .42.…”

“…Therefore, the learning score based on pattern-random difference likely underestimates the true learning that occurs in the task. Furthermore, unlike Buffington et al (2021), we excluded trials that were repetitions (e.g., 222) or trills (e.g., 232), as subjects likely have pre-existing response tendencies to such trials (Soetens et al, 2004). This also possibly increased the robustness of our learning scores.…”

Prediction and Error-Based Learning in L2 Processing and Acquisition

“…More research is needed to investigate the complex relationships between these factors. The role of cognitive and linguistic individual differences, is clearly a burgeoning area of interest in the language learning sciences (e.g., Bolibaugh & Foster, 2021;Buffington et al, 2021;Pili-Moss, 2021;Riches & Jackson, 2018;Walker et al, 2020; including special issues dedicated to the topic such as those edited by Andringa & Dąbrowska, 2019;Roberts & Meyer, 2012), and investigating the role of individual cognitive abilities in prediction and, specifically, error-based learning, in L1 and L2 (or Lx) speakers would constitute a timely extension of this agenda. Such research could shed more light on some of the factors we have reviewed, such as the varying effect of L2 proficiency (e.g., helping to clarify the relative contributions of knowledge representation and "procedural" proficiency in enabling prediction).…”

Long-term memory predictors of adult language learning at the interface between syntactic form and meaning

“…Procedural learning ability was indexed by performance on an alternate serial reaction time task adapted from a previously employed serial reaction time paradigm [ 40 ]. Previous studies investigating the reliability of this task have found a test-retest r = .45 [ 41 ], Cronbach α values between .75 and .79 (RT scores) and between .69 and .75 (accuracy scores) [ 42 ], as well as a Spearman-Brown split-half reliability index of .42 [ 25 ]. Participants saw four squares arranged in a diamond shape configuration on screen and were asked to immediately react to the changes in the position of a smiley by pressing the corresponding buttons on a game controller.…”

“…Overall the results of these studies have converged in finding that declarative learning abilities (assessed via behavioural verbal and visual declarative memory measures) tended to positively predict learning of morphosyntax at early stages of exposure (see also Hamrick [ 24 ] for similar results obtained using a different artificial language paradigm). However, after extended exposure (or after periods of no exposure [ 24 ]) morphosyntactic learning (particularly word order) became positively related to learners’ procedural learning ability measured by probabilistic learning tasks or implicit sequence learning tasks [ 21 – 23 , 25 ].…”