Comparability of educational achievement and learning attitudes across nations

Abstract

We estimated the invariance of educational achievement (EA) and learning attitudes (LA) measures across nations. A multi-group confirmatory factor analysis was used to estimate the invariance of educational achievement and learning attitudes across 55 nations (Programme for International Student Assessment [PISA] 2006 data, N = 354,203). The constructs had the same meaning (factor loadings) but different scales (intercepts). Our conclusion is that comparisons of the relationships between educational achievement and learning attitudes across countries need to take into consideration two sources of variability: individual differences of students and group differences of educational systems. The lack of scalar invariance in EA and LA measures means that the relationships between EA and LA may have a different meaning at the level of nations and at the student level within countries. In other words, as PISA measures are not invariant in scalar sense, the comparisons across countries with nationally aggregated scores are not justified.

Malaysian and Singaporean students’ affective characteristics and mathematics performance: evidence from PISA 2012

“…The items of students’ affective characteristics might be too general and therefore might not able to fully capture their perception of affective characteristics. In addition, it is rather difficult to have the content of each item being perceived and interpreted in the same way across groups from different contexts and cultural background (Täht and Must 2013). In relation to this, measurement invariance approach could be employed to identify the items that convey the same meaning across groups prior to conduct the analyses between countries (Byrne 2008).…”

Effects of Anchoring Vignettes on Comparability and Predictive Validity of Student Self-Reports in 64 Cultures

“…In PISA, meticulous design and implementation have lent much confidence in the comparability of the scales; yet, formal statistical testing of scale comparability has not been formally reported in previous cycles (OECD, 2013). Some attempts to establish measurement invariance in PISA reported limited comparability (e.g., Täht & Must, 2013). Three main levels of comparability (also called invariance) in scales can be distinguished and statistically tested: (a) Configural invariance means across cultures, the construct is understood as the same.…”

A Mixed Methods Study of the Effects of Clicker Use on Math Anxiety and Achievement in Mathematics

“…Educational researchers and policymakers have expressed great concern about mathematics achievement in the United States based on comparisons with other countries (Kankaraš & Moors, 2014;Täht & Must, 2013;Tienken, 2013;Turgut, 2013;Zhao, 2005).…”

“…Results from the Program for International Student Assessment (PISA) and Trends in International Mathematics and Science Study (TIMSS) suggest that numerous countries are outperforming the United States in mathematics (Täht & Must, 2013;Zhao, 2005). In 2010, Secretary of Education Arne Duncan stated, "Unfortunately, the 2009 PISA results show that American students are poorly prepared to compete in today's knowledge economy.…”

“…The authors concluded that the comparison of test results across countries is highly questionable. Täht and Must (2013) conducted a factor analysis based on the PISA 2006 data. They concluded that comparisons of different countries' PISA scores are unjustified.…”