The Use of Generalizability (G) Theory in the Testing of Linguistic Minorities

Abstract

We contend that generalizability (G) theory allows the design of psychometric approaches to testing English‐language learners (ELLs) that are consistent with current thinking in linguistics. We used G theory to estimate the amount of measurement error due to code (language or dialect). Fourth‐ and fifth‐grade ELLs, native speakers of Haitian‐Creole from two speech communities, were given the same set of mathematics items in the standard English and standard Haitian‐Creole dialects (Sample 1) or in the standard and local dialects of Haitian‐Creole (Samples 2 and 3). The largest measurement error observed was produced by the interaction of student, item, and code. Our results indicate that the reliability and dependability of ELL achievement measures is affected by two facts that operate in combination: Each test item poses a unique set of linguistic challenges and each student has a unique set of linguistic strengths and weaknesses. This sensitivity to language appears to take place at the level of dialect. Also, students from different speech communities within the same broad linguistic group may differ considerably in the number of items needed to obtain dependable measures of their academic achievement. Whether students are tested in English or in their first language, dialect variation needs to be considered if language as a source of measurement error is to be effectively addressed.

Generalizability theory and the fair and valid assessment of linguistic minorities

“…Testing students with the same set of items in two languages is key to investigating how language variation affects the validity of measures of academic achievement for ELLs (Solano-Flores & Li, 2006, 2009a, 2009b. Notice, however, that this approach differs from research and practice in ELL testing, which has used two-sided, L1 and L2 versions of the same tests.…”

“…To examine more carefully score variation due to language and also to determine if dialect is an important source of measurement in the testing of linguistic minorities, we conducted a series of studies using the same methods described in the section above and new samples of Grade 4 and Grade 5 ELLs who were native speakers of Haitian-Creole (SolanoFlores & Li, 2006) and native speakers of Spanish (Solano-Flores & Li, 2009b). We created a mathematics test with NAEP released items.…”

Capacidad evaluativa, validez cultural, y validez consecuencial en PISA

“…Actualmente los estudios de generalizabilidad no forman parte de la práctica establecida para examinar aspectos de cultura, aunque la investigación muestra su importancia en el análisis de la validez cultural (Solano-Flores & Li, 2006, 2009. La teoría de la generalizabilidad (G) (Brennan, 2001;Cronbach, Gleser, Nanda & Rajaratnam, 1972;Shavelson & Webb, 1991) es una teoría del error de medición y también del muestreo de observaciones (Kane, 1982).…”

“…La implicación de este hallazgo lleva a otra condición -Desagregación de los Datos-. Desagregar los datos por grupos culturales y comparar los grupos en términos de la calidad psicométrica de sus puntuaciones en la prueba es una estrategia más rigurosa que simplemente comparar grupos en términos de las medias de sus calificaciones (Solano-Flores & Li, 2013).…”

Interpretaciones no intencionadas e intencionadas y usos de los resultados de PISA : Una perspectiva de validez consecuencial

“…Generalizability studies are not currently part of established practices related to addressing culture, though research shows it is relevant to examining cultural validity (Solano-Flores & Li, 2006, 2009. Generalizability (G) theory (Brennan, 2001;Cronbach, Gleser, Nanda & Rajaratnam, 1972;Shavelson & Webb, 1991) is a theory of measurement error and also a theory of sampling of observations (Kane, 1982).…”