Changes in achievement on PISA: the case of Ireland and implications for international assessment practice

Abstract

The PISA 2009 results for Ireland indicated a large decline in reading literacy scores since PISA 2000 (the largest of 38 countries). The decline in mathematics scores since PISA 2003 was the second largest of 39 countries. In contrast, there was no change in science achievement since PISA 2006. These results prompted detailed investigations into possible reasons for the declines, particularly in reading. This paper considers the changes in achievement observed for Ireland in PISA 2009 under two themes: implementation of PISA in Ireland and changes in the cohort of students participating in PISA, and response patterns on the PISA test (as measures of student engagement). It is argued that the case of Ireland represents the 'perfect storm', since a range of factors appear to have been in operation to produce the results. The discussion attempts to show how the case of Ireland can be relevant to other countries which may have experienced changes in PISA test scores over time. Some of the findings have relevance to international practice in large-scale surveys of educational achievement more generally. ReviewWhen first published, the results for PISA 2009 reading, and to a lesser extent mathematics, attracted media attention and commentary in Ireland. For example, the Irish Times discussed the results under the headline "shattering the myth of a world-class education system" (December 8, 2010), while the Irish Independent noted that "There was shock last year when it emerged there was a fall in reading and maths scores for Irish students in the PISA" (April 3, 2012). Education Matters described the results as "an urgent call to action" (December 14, 2010). Ireland's mean reading score on PISA showed the largest decline since 2000 across the 38 countries for which results could be compared (31 score points, or close to one-third of an international standard deviation), and mathematics showed the second-largest decline since 2003 across the 39 countries that could be compared (16 points, or one-sixth of an international standard deviation). In contrast, achievement in science remained stable (OECD, 2010a). These patterns of results were unexpected given the absence of other evidence of a decline in educational standards in Ireland.

Commentary: On the Importance of the Speed-Ability Trade-Off When Dealing With Not Reached Items

“…The first assumption is unlikely to hold in many applications. The second assumption may hold in high stakes assessments, while in low stakes assessments, for which the missing data approaches have been suggested, empirical data (e.g., Cosgrove and Cartwright, 2014; Pohl et al, under review); suggest otherwise. Also note that this solution requires moving from measuring optimal ability for a given time limit and instead opt for measuring effective ability given the chosen speed.…”

Dealing With Item Nonresponse in Large‐Scale Cognitive Assessments: The Impact of Missing Data Methods on Estimated Explanatory Relationships

“…Fixing it at 0 resulted in approximately 50% missing data, which we chose in order to enhance the effects. Changing the mean to 2.5 resulted in a realistic amount of approximately 10% missing data (see, e.g., Cosgrove & Cartwright, 2014;Koretz et al, 1993;OECD, 2012). The means of θ and Z were set to 0 in both conditions.…”

When Nonresponse Mechanisms Change: Effects on Trends and Group Comparisons in International Large-Scale Assessments

“…From the results and from additional analyses by Cartwright (2011), they deduced that nonresponses are distinct from proficiency and that they are highly nationally idiosyncratic characteristics that change over time. Overall, Cosgrove and Cartwright's (2014) findings suggest that missing values, especially changes in the percentages of missing values and changes in the missing data mechanisms, may have affected Ireland's decline in reading achievement. However, the impact of the detected changes in the characteristics remained unclear.…”

“…Furthermore, the increase in missing values and the decrease in correct responses were observed to be larger for the linking items than for the new items. Cosgrove and Cartwright (2014) hypothesized that a general decline in student engagement in Ireland in 2009 may have resulted in a change in the missing data mechanism. To examine changes in the average proportion of missing values, they calculated time-series correlations for the average proportion of missing responses from 2003 to 2009 for Ireland and for all OECD countries.…”