Probing the Internal Validity of the LLAMA Language Aptitude Tests

Abstract

Over the past decade, the LLAMA language aptitude test battery has come to play an increasingly important role as an instrument in research on individual differences in language development. However, a potentially serious problem that has been pointed out by several scholars is that the LLAMA has not yet been carefully validated. We addressed this issue by examining the internal validity of this test battery. We collected LLAMA data from 350 participants and assessed these data using classical item analysis, Rasch analysis, and principal component analysis within a framework of best practices in educational and psychological test validation. The results show that only one out of the four subtests (LLAMA B) produced scores that fit a latent trait model with sufficient accuracy. This suggests that researchers using the LLAMA battery must treat their results with appropriate carefulness and also that there is potential for refining the LLAMA further.

Language Aptitude and Crosslinguistic Influence in Initial L2 Learning

“…The internal consistency (coefficient alpha) of the scores was 0.80 (LLAMA B), 0.55 (LLAMA D), 0.80 (LLAMA E) and 0.67 (LLAMA F). The lower alpha values for LLAMA D and F were expected, and in line with other studies that have used the LLAMA (for a comprehensive discussion of reliability issues in the LLAMA tests, see Bokander & Bylund, 2020).…”

“…Language aptitude was measured with the LLAMA test battery (Meara, 2005), which has become a popular measure of language aptitude during the past decade. Several aspects of its validity have been investigated with speakers of a wide range of L1s (Bokander & Bylund, 2020;Granena, 2013a;Rogers et al, 2017) and, with importance for the present study, LLAMA scores do not seem to be influenced by the test takers' L1 background. Some of the LLAMA subtests have tended to produce low internal consistency estimates (typically resulting in wider confidence intervals for correlations with other variables, i.e., low power to detect aptitude related effects).…”

“…Some of the LLAMA subtests have tended to produce low internal consistency estimates (typically resulting in wider confidence intervals for correlations with other variables, i.e., low power to detect aptitude related effects). However, in many studies (see Bokander & Bylund, 2020, for a comprehensive review), the LLAMA subtests have indeed demonstrated expected effects of language aptitude, thus implying that LLAMA may serve as a useful aptitude measure given that findings based on this instrument are interpreted with due carefulness.…”

The Roles of Recasts, Task Complexity, and Aptitude in Child Second Language Development

“…It includes measures of the central executive component of working memory, phonological short‐term memory, associative memory, long‐term memory retrieval, processing speed, implicit learning, and auditory discrimination. The present study used the LLAMA test (Meara, ) as a measure of aptitude (but see Bokander & Bylund, , for reservations about the validity of LLAMA as a test of aptitude). The LLAMA test, an instrument frequently employed to test aptitude in current L2 acquisition research, is composed of four subtests.…”

The methodology of the research on language aptitude: A systematic review

“…The test battery is comprised of four subtests: LLAMA_B, _D, _E, and _F. Among the four subtests, LLAMA_B, _E, and _F have been found to be separate from LLAMA_D, and when the four subtests load onto the same factor, LLAMA_D shows the lowest factor loading, suggesting that its correlation with the factor is weakest (Bokander & Bylund, 2020;Granena & Long, 2013;Li & Qian, in press). Furthermore, LLAMA_D has also demonstrated attributes of implicit aptitude, although its validity as a test of implicit aptitude has been questioned.…”