Concerns about how to ensure the valid and equitable assessment of English-language learners (ELLs) and other students from culturally non-mainstream backgrounds are longstanding. This article proposes that new paradigms in the research and practice related to ELL testing are needed to address the complexities of language and culture more effectively. Three main areas are identified as key to this paradigm shift: test review, test development, and treatment of language as a source of measurement error. Research examples are provided that illustrate that the proposed paradigm shift is not only necessary but also possible. The authors propose the combined use of generalizability theory and research designs in which ELLs are given the same items in both English and their native languages—an approach that has the potential to reveal more fine-grained understandings of the interactions among first and second language profi-ciency, student content knowledge, and the linguistic and content demands of test items.

We propose the concept of cultural validity as a form of test validity in science assessment. The conceptual relevance of cultural validity is supported by evidence that culture and society shape an individual's mind and thinking. To attain cultural validity, the process of assessment development must consider how the sociocultural context in which students live influences the ways in which they make sense of science items and the ways in which they solve them. These sociocultural influences include the values, beliefs, experiences, communication patterns, teaching and learning styles, and epistemologies inherent in the students' cultural backgrounds, as well as the socioeconomic conditions prevailing in their cultural groups. We contend that current approaches to handling student diversity in assessment (e.g., adapting or translating tests, providing assessment accommodations, estimating test cultural bias) are limited and lack sociocultural perspective. We find that attaining cultural validity may conflict with current basic principles and assumptions in testing, such as item independence and standardization. We discuss the ways in which adopting cultural validity as a criterion for test validity makes it necessary to shift assessment paradigms and adopt new procedures for assessment development. © 2001 John Wiley & Sons, Inc. J Res Sci Teach 38: 553–573, 2001

The testing of English language learners (ELLs) is, to a large extent, a random process because of poor implementation and factors that are uncertain or beyond control. Yet current testing practices and policies appear to be based on deterministic views of language and linguistic groups and erroneous assumptions about the capacity of assessment systems to serve ELLs. The question Who is given tests in what language by whom, when, and where? provides a conceptual framework for examining testing as a communication process between assessment systems and ELLs. Probabilistic approaches based on generalizability theory—a psychometric theory of measurement error—allow examination of the extent to which assessment systems’ inability to effectively communicate with ELLs affects the dependability of academic achievement measures.

No abstract

This article examines the intersection of psychornetrics and sociolinguists in the testing of English language learners (ELLs); it discusses language, dialect, and register as sources of measurement error. Research findings show that the dialect of the language in which students are tested (e.g., local or standard English) is as important as language as a facet that influences score dependability in ELL testing. The development, localization, review, and sampling of items are examined as aspects of the process of test construction critical to properly attaining linguistic alignment: the correspondence between the features of the dialect and the register used in a test, and the features of the language to which ELLs are exposed in both formal and instructional contexts.