Comparing OECD PISA Reading in English to Other Languages: Identifying Potential Sources of Non-Invariance

Asil, Mustafa; Brown, Gavin

doi:10.1080/15305058.2015.1064431

Cited by 34 publications

(25 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The studies also found that the magnitude of DIF increased when comparing language versions administered in countries with different GDP per capita. These findings were supported by Asil and Brown's (2016) analyses of PISA 2009 reading tests. In addition, Kreiner and Christensen (2014) detected significant variations in the way some items behaved in language versions of PISA 2006 reading survey and claimed that deleting 'biased' items would have serious implications on the reliability of country rankings.…”

Section: Introductionsupporting

confidence: 52%

“…Despite the rigorous quality control involved in the translation and adaptation processes of ILSAs, numerous studies have highlighted how items may behave differently across language versions and countries (Asil & Brown, 2016;Ercikan & Koh, 2005; A Grisay, de Jong, Gebhardt, Berezner, & Halleux-Monseur, 2007;Aletta Grisay & Monseur, 2007;Huang, Wilson, & Wang, 2016;Kreiner & Christensen, 2014;Oliveri & Ercikan, 2011;Sandilands, Oliveri, Zumbo, & Ercikan, 2013;Wu & Ercikan, 2006). These studies used statistical techniques generally referred to as differential item functioning (DIF) analyses to investigate potential bias across language versions within countries (e.g.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Language effects in international testing: the case of PISA 2006 science items

Masri

Baird

Graesser

2016

Assessment in Education: Principles, Policy & Practice

View full text Add to dashboard Cite

We investigate the extent to which language versions (English, French and Arabic) of the same science test are comparable in terms of item difficulty and demands. We argue that language is an inextricable part of the scientific literacy construct, be it intended or not by the examiner. This argument has considerable implications on methodologies used to address the equivalence of multiple language versions of the same assessment, including in the context of international assessment where cross-cultural fairness is a concern. We also argue that none of the available statistical or qualitative techniques are capable of teasing out the language variable and neutralising its potential effects on item difficulty and demands. Exploring the use of automated text analysis tools at the quality control stage may be successful in addressing some of these challenges.

show abstract

Section: Introductionsupporting

confidence: 52%

Section: Introductionmentioning

confidence: 99%

Language effects in international testing: the case of PISA 2006 science items

Masri

Baird

Graesser

2016

Assessment in Education: Principles, Policy & Practice

View full text Add to dashboard Cite

show abstract

“…While in some studies, no invariance was provided in any model (Gülleroğlu, 2016), in some it was observed that only configural invariance was provided (Hansson & Gustafsson, 2013;Sırgancı & Çakan, 2020). In some of them, it was determined that metric invariance was hold (Asil & Brown, 2015;Pauwels, 2018;İmrol, 2017;Luo, 2010). It has been reported that scalar invariance is held in a few studies (Uzun & Öğretmen, 2010) and rare of the studies, strict invariance is provided (Wu, Li & Zumbo (2007).…”

Section: Discussionmentioning

confidence: 99%

Measurement Invariance Testing with Alignment Method: Many Groups Comparison

Sırgancı

Uyumaz

Yandı

2020

International Journal of Assessment Tools in Education

View full text Add to dashboard Cite

It is necessary to examine the measurement invariance (MI) among groups in studies where different groups are compared by using a measurement instrument. Most of the studies, measurement invariance is tested with multiple group confirmatory factor analysis. This model applies many model adjustments based on the modification indexes. Therefore, it is not practical due to too many large modification indexes while testing MI over many groups. Besides scalar model is a poor model fit when comparing many groups and so does not hold MI. In this study, the aim is to explain the basic concepts and processes of the alignment method which is offered as a new method for testing MI and illustrate an application on the real data set. In this study, measurement invariance among 56 countries including Turkey is tested with alignment method in order to set an example for researchers. For this purpose, the Instrumental Motivation Scale data, which is one of the psychological measurement instruments used in PISA 2015, was used. As a result of MG-CFA, it was found that configural invariance was ensured. The fit indexes of CFI and TLI were calculated as 0.982 and 0.946 respectively in this stage. After that, metric invariance was tested by considering the difference of fit indices obtained for the two stages. It was found that the metric invariance could not be provided. Alignment results show which countries hold MI and which do not. Besides it provides information which items have the most invariants for groups that hold MI.

show abstract

“…Reading skills are not only used to acquire knowledge in schools but more importantly are a necessary ability which guides individuals through their whole life (Motiejunaite, Noorani, & Monseur, 2014). But linguistic patterns, structures and the processes of their acquisition are very different across languages and the assessment of reading in an international context has also triggered critical examinations of the underlying cognitive-psychological concepts associated with the learning processes (Asil & Brown, 2016;Evans & Levinson, 2009;Grisay, Gonzalez, & Monseur, 2009). Language influences on item difficulties and test results have been documented for subjects such as science as well (e.g.…”

Section: Doing Research With International Assessment Studies: Methodmentioning

confidence: 99%