The Validity of Administering Large-Scale Content Assessments to English Language Learners: An Investigation From Three Perspectives

Abstract

Data from a large public school district (referred to as Site 1 from this point on) for Grades 2 through 8 for the 1999 student population were analyzed for all students including English language learners (ELLs). The data included student responses to the reading and mathematics subtests of the Iowa Tests of Basic Skills 2 (ITBS) and student background data such as race, gender, birth date, and number of years of participation in a bilingual education program (number of years of bilingual service). Descriptive statistics and the percent of over-achievement of non-ELL students over ELL students were computed and compared across the different subtest content areas. In multiple regression analyses, student English learning status was related to student test scores and background variables. A state department of education (referred to as Site 2 from this point on) provided us with student background data and item-level data on the Stanford Achievement Test Series, Ninth Edition (Stanford 9) 3 for all students in Grades 2 through 11 who were enrolled in the public schools statewide for the 1997-1998 academic year. Descriptive statistics compared ELL and non-ELL student performance by subgroup and across the different content areas. In a canonical correlation model the relationship between student language proficiency level, parent education, and family socioeconomic status (SES) (the Set 2 variables) and Stanford 9 performance (the Set 1 variables) was examined.

Effects of linguistic complexity and accommodations on estimates of ability for students with learning disabilities

“…Although these effects may not be consistent across all items, in aggregate on an assessment, the effects of LC and accommodations may form a different test for students with diverse language backgrounds. Abedi et al (2005) found a difference in the internal consistency of test scores for ELL students versus non-ELL students; LC and accommodations may lead to similar results for SLD. Although the reasons for challenges with high LC are different for both ELL students and SLD, high LC may be a potential obstacle to optimal performance on assessment items (and particularly, reading items) for both groups.…”

“…Research has shown that unnecessarily complex language (e.g., uncommon vocabulary, words with multiple meanings, or complicated word clauses) may interfere with a student's ability to exhibit content knowledge on a standardized assessment and that this interference can hinge on student English language proficiency (e.g., Abedi, Hofstetter, & Lord, 2004;Abedi & Lord, 2001;Abedi et al, 2005). For example, it has been shown that ELL students have a more difficult time than their non-ELL student peers accurately answering questions with a high discourse demand, or language processing beyond the sentence level (Abedi et al, 2005). In addition to the effects of accommodations, the concept of language demand of assessment items is an important consideration when interpreting the scores of students with potential language processing challenges.…”

Linguistic Complexity, Schematic Representations, and Differential Item Functioning for English Language Learners in Math Tests

“…For example, items with ambiguous or multiple meaning words were more difficult than items with fewer or none of these features for both ELLs and non-ELLs in the fourth-grade KGMA, but this effect was not significant in the seventh and tenth grades (Shaftel et al, 2006). Likewise, the presence of idioms, uncommon words and/or words used in an uncommon way was not significantly correlated with (unconditional) item p value differences across ELLs and non-ELLs in a third-grade math subtest of a standardized achievement test (Abedi et al, 2005). Neither was the presence of low-frequency vocabulary in the third and eighth grade SAT9 (Lord et al, 2000).…”

“…The presence of complex or uncommon syntactic structures did not show significant correlations with group p value differences in the third-grade math subtest, whereas the presence of multiple clauses did (r D .32, p D .02; Abedi et al, 2005). Neither the number of passive voice sentences nor the number of subordinate clauses has shown significant effect in any of the tests and grades studied (NAEP Grade 8 in Abedi et al, 1997; SAT9 Grades 3 and 8 in Lord et al, 2000;KGMA Grades 4, 7, and 10 in Shaftel et al, 2006).…”

“…In the last decade, a few important empirical studies have investigated the relationship between some of these linguistic features and the difficulty (p value) of math word problems for ELLs and non-ELLs in elementary, middle, and high school levels (Abedi et al, 2005;Abedi & Lord, 2001;Abedi et al, 1998;Abedi et al, 1997;Lord, Abedi, & Poosuthasee, 2000;Shaftel, Belton-Kocher, Glasnapp, & Poggio, 2006). Although many of these studies did, as predicted, find a relationship between linguistic complexity and ELLs' performance in math word problems, the effect of specific linguistic features varied from test to test and from one grade to another.…”