Do Different Approaches to Examining Construct Comparability in Multilanguage Assessments Lead to Similar Conclusions?

Abstract

In this study, we examine the degree of construct comparability and possible sources of incomparability of the English and French versions of the Programme for International Student Assessment (PISA) 2003 problem-solving measure administered in Canada. Several approaches were used to examine construct comparability at the test-(examination of test data structure, reliability comparisons and test characteristic curves) and item-levels (differential item functioning, item parameter correlations, and linguistic comparisons). Results from the test-level analyses indicate that the two language versions of PISA are highly similar as shown by similarity of internal consistency coefficients, test data structure (same number of factors and item factor loadings) and test characteristic curves for the two language versions of the tests. However, results of item-level analyses reveal several differences between the two language versions as shown by large proportions of items displaying differential item functioning, differences in item parameter correlations (discrimination parameters) and number of items found to contain linguistic differences.In Canada, most large-scale assessments are administered in its two official languages: English and French. Performance data from such assessments are used to make comparisons of student achievement, to inform curriculum, program development and evaluation, and to make decisions concerning educational policies. Implicit assumptions made in using results from these assessments are that measurements are comparable and that they are based on the same construct for the two language groups. The validity of uses of and inferences made based upon results from these assessments critically depend upon their comparability.

Effects of Population Heterogeneity on Accuracy of DIF Detection

“…It might also contribute to challenges in identifying sources of DIF documented in previous research (Ercikan et al, 2010;Oliveri & Ercikan, 2011). Multiple studies have been conducted with gender, ethnic, and language comparison groups to improve DIF detection and the identification of its sources by creating more homogeneous groups in measurement comparability research.…”

“…The identification of differential item functioning (DIF) may signal that an item is measuring construct-irrelevant factors such as differential familiarity with item types, formats, or vocabulary knowledge for one or more of the comparison groups (Ercikan, Gierl, McCreith, Puhan, & Koh, 2004;Ercikan & Lyons-Thomas, 2013;Oliveri & Ercikan, 2011). Accurate DIF detection is central to making claims regarding whether an item should be used in an assessment or whether modification is required in order to reduce or eliminate construct-irrelevant variance across comparison groups.…”

Measurement Comparability of Reading in the English and French Canadian Populations: Special Case of the 2011 Progress in International Reading Literacy Study

“…Items are typically flagged for DIF if response probabilities for examinees at the same ability levels depend on group membership. As different methods for identifying DIF may not give identical results, the use of more than one method is recommended, to allow for the corroboration of DIF status for the items analyzed (Ercikan and McCreith, 2002;Oliveri and Ercikan, 2011). In this research, an IRT-based approach and logistic/ordinal logistic regression approaches were used.…”

“…It is incumbent on countries with multiple official languages to take reasonable steps to ensure that linguistic groups are given the opportunity to perceive and respond to tests in the same way (Fairbairn and Fox, 2009;Rogers et al, 2010;Marotta et al, 2015). Yet, research conducted in Canada comparing the French and English versions of LSAs has found that 18-60% of items function differentially for the two groups (Gierl et al, 1999;Gierl, 2000;Ercikan and McCreith, 2002;Ercikan et al, 2004b;Oliveri and Ercikan, 2011;Marotta et al, 2015).…”

A Framework for Developing Comparable Multilingual Assessments for Minority Populations: Why Context Matters

“…To illustrate, reading competencies may develop at different speeds and in different ways across languages with differing alphabets (Spielberger, Moscoso, & Brunner, 2005). Additional factors that might lead to the development of tests with limited comparability include natural variation in difficulty, commonality, or contextual meaning of vocabulary and differential sentence length or complexity (Hambleton, Merenda, & Spielberger, 2005;Oliveri & Ercikan, 2011;Solano-Flores, Backhoff, & ContreraNiño, 2009). Several guidelines have been developed to address comparability issues arising in test adaptation.…”