PISA Data: Raising concerns with its use in policy settings

Abstract

This article considers the role played by policy makers, government organisations, and research institutes (sometimes labelled "think tanks") in the analysis, use and reporting of PISA data for the purposes of policy advice and advocacy. It draws on the ideas of Rizvi and Lingard (2010), Bogdandy and Goldmann (2012) and others to explore the ways in which such "agents of change" can interpret, manipulate and disseminate the results of data arising from large scale assessment survey programs such as PISA to influence and determine political and/or educational research agendas. This article illustrates this issue by highlighting the uncertainty surrounding the PISA data that have been used by a number of prominent, high profile agents of change to defend policy directions and advice. The final section of this paper highlights the need for policy makers and their advisors to become better informed of the technical limitations of using international achievement data if such data are to be used to inform policy development and educational reforms.

A circulação global de políticas de alfabetização: o método fônico, desigualdade e movimentos políticos neoconservadores

“…This pattern of a "crisis" in education standards, focused on disadvantaged populations, providing the pretext for neoliberal reforms has been repeated in other national settings, including Japan (TAKAYAMA, 2007), and Australia (WINDLE, 2011). This repetition can also be observed on other scales, locally in systems like Chicago's (LIPMAN, 2004), and transnationally in the influence of the OECD (GILLIS et al, 2016). National-level policy coalition formation in the USA appears to be the originating model for Phonics Policy Advocacy, with its key elements present to varying degrees in other settings and across scales.…”

PISA, policymaking and political pantomime: education policy referencing between England and Hong Kong

“…The data and the accompanying analysis are framed to facilitate the identification of specific key factors (such as a demanding curriculum, good quality teachers and a high degree of autonomy for school principals) that are claimed to bring about strong PISA scores, which in turn are associated with strong economic productivity. This approach requires a dash of fairy dust: in the absence of an ability to establish a causal connection between the input factors identified and the desired outputs (Auld and Morris 2016;Forestier 2015;Gillis, Polesel, and Wu 2016) it relies on speculation, impressions, correlations and assertions to fill the vacuum. In Hong Kong, in contrast, overseas academics and specialists in specific areas are brought in to synthesise their expertise/support implementation with that of local stakeholders.…”

Introduction

“…The Grattan Institute, a centrist 'academic' think tank affiliated with the University of Melbourne, has exerted particular skill in this regard, generating widespread media coverage when members of its 'School Education' program release new reports (see Loughland and Thompson 2015;Gillis et al 2015). Lingard (2015) explores the work of the Centre for Independent Studies, and Reid (2015) examines the Education Standards Institute, a Melbourne-based education think tank established by Dr Kevin Donnelly and committed to a liberal view of education, and Christian beliefs and values.…”

“…Another example of this, canvassed in several papers in this issue, is the uncritical and invalid use of test data from a variety of tests including PISA and NAPLAN. As Gillis et al (2015) argue, this is particularly true regarding PISA data, where 'the uncertainty surrounding the use of large scale assessment surveys… are not always taken into consideration (either by accident or by design) when used by prominent, high profile agents of change for policy reform purposes'. Their argument is that when these data are used in invalid ways, which pursue pre-existing agendas, think tanks obscure the usefulness of the data.…”

“…In doing so, however, they manage to operate as 'highly visible players on the policy scene, issuing studies aimed at politicians and the wider public, hosting symposia, press conferences, and political speeches, and offering a 'government in exile' for sidelined officials awaiting a return to public office' (Medvetz 2012, p. 118). Several papers in this issue examine the implications of think tank or edu-business involvement in relation to shifts towards large-scale testing, standardisation and big data (see Gillis et al 2015;Loughland and Thompson 2015;Savage 2015;Lingard 2015;Hogan 2015). A second critical aspect of the think tank/edu-business story in Australia concerns the rise of new forms of expertise, located outside the traditional institutions of educational expertise, such as government bureaucracies, schools and academia.…”