Generalizability theory and the fair and valid assessment of linguistic minorities

Abstract

We discuss generalizability (G) theory and the fair and valid assessment of linguistic minorities, especially emergent bilinguals. G theory allows examination of the relationship between score variation and language variation (e.g., variation of proficiency across languages, language modes, and social contexts). Studies examining score variation across items administered in emergent bilinguals' first and second languages show that the interaction of student and the facets (sources of measurement error) item and language is an important source of score variation. Each item poses a unique set of linguistic challenges in each language, and each emergent bilingual individual has a unique set of strengths and weaknesses in each language. Based on these findings, G theory can inform the process of test construction in large-scale testing programmes and the development of testing models that ensure more valid and fair interpretations of test scores for linguistic minorities.

Language Shift and the Inclusion of Indigenous Populations in Large-Scale Assessment Programs

“…Bilingual populations are tremendously heterogeneous (Valdés & Figueroa, 1994;Oller, Pearson, & Cobo-Lewis, 2007). Due to this heterogeneity, the amount of measurement error due to the interaction of student and item may be larger than that typically observed for mainstream populations (Solano-Flores & Li, 2013). Language shift appears to be an additional source of performance instability that poses an even a greater challenge for validly testing indigenous populations.…”

“…G theory has been proven useful in the testing of linguistically diverse populations because it allows examination of language as a source of measurement error and, more specifically, the amount of measurement error due to the interaction of student, item, and the language in which tests are administered (Solano-Flores & Li, 2013). Unlike conventional approaches used in the testing of linguistic minorities, a G theory-based approach focuses on the dependability of test scores, rather than performance differences with a reference group (Solano-Flores & Li, 2013).…”

“…A case in point is the use of testing accommodations-modifications of tests intended to ensure the students' access to the content of test items (see Abedi, Hofstetter, & Lord, 2004). However, there is a limit to the extent to which these actions can minimize limited proficiency in the language of testing as a threat to the validity of interpretations of test scores (Pennock-Roman & Rivera, 2011;Solano-Flores & Li, 2013). Important validity issues in the testing of linguistic minorities are yet to be resolved (Ercikan, Roth, Simon, Sandilands, & Lyons-Thomas, 2014;Sireci & Allalouf, 2003).…”

An Evaluative Framework for Reviewing Fairness Standards and Practices in Educational Tests

“…The 2014 Standards also encourage the use of validity studies such as cognitive labs with special populations when sample sizes are not large enough for DIF studies. Alternate methodologies for DIF such as Solano‐Flores and Li's () use of generalizability theory for examining the interaction between items, students, and difference sources of variance beyond score variation and language variation for ELLs and non‐ELLs might also be a methodology that could provide additional insights into valid interpretation and use for these subgroups.…”

Corrigiendo las diferencias de uso de escala entre países de América Latina, Portugal y España en PISA

“…The implication of this finding brings to another Practices conditionDisaggregation of Data. Disaggregating data by cultural group and comparing groups in terms of the dependability of the scores they obtain in a test is a more rigorous approach than comparing groups as to their mean test scores (Solano-Flores & Li, 2013).…”