Achievement tests from an item perspective: An exploration of single item data from the PISA and TIMSS studies, and how such data can inform us about students’ knowledge and thinking in science

Abstract

The thesis presents secondary analyses of data from a large-scale international assessment of students' achievement in mathematics, reading and science entitled the Programme for International Student Assessment (PISA) which is initiated and monitored by the OECD. The thesis has been conducted and supervised at the Department of Teacher Education and School Development at the Faculty of Education, University of Oslo. The work presented in the dissertation is framed by two overarching questions relating specifically to the science achievement data in PISA: What is the nature of the information in the cognitive science items in PISA?, and; what is the analytical potential of this information? These two questions have been used as guides in several explorative analyses of the science achievement data in PISA presented in three published papers. The background for posing these general questions is the fact that when the achievement data are reported from these studies all the item specific information has been lost: Initially students' responses to each of the items have been marked by a number of specific codes capturing qualitative aspects of the responses. These codes are subsequently reduced or aggregated into credits or score points. Then the item scores are aggregated into one (or a few) reliable and valid overall achievement scores. The first paper is primarily an introduction into this general background and a rationale is developed for why the study of the single items before they are aggregated is relevant and possible. The second paper uses homogeneity analysis in the study of the codes used in the initial marking before they are reduced to score points. The third paper studies the relative achievement profile across all items and across all participating countries by applying cluster analysis. The analysis demonstrates that countries are clustered and each cluster has a distinct profile across the items, and these profiles are not captured in the overall achievement scores. One overall message in the thesis is that large-scale international comparative achievement studies do not only provide high-quality data of students' overall science achievement across the world. There is also a fine-structure in the data across the single items. It is envisaged that these data may also be used in secondary analyses targeting more specific research questions within the research field of science education across the world.

Profiles of Students’ Interest in Science Issues around the World: Analysis of data from PISA 2006

“…These properties of a cluster can be referred to as internal cohesion and external isolation (Olsen, 2005a). Cluster analysis is a generic term for methods aiming to cluster individual cases (or variables) into larger groups, which at the same time are similar to cases within the group and dissimilar to cases outside the group.…”

Å forstå det man leser – å trekke slutninger i skjønnlitteraturog sakprosa

“…Analysemetoden er en såkalt residualanalyse der et lands prestasjoner på oppgaver korrigeres for både landets gjennomsnittsskår og oppgavens gjennomsnittlige vanskegrad (om metoden, se f.eks. Olsen, 2005). Vi har tatt utgangspunkt i nasjonal løsningsprosent for alle leseoppgavene i prøven (N = 242).…”

Untitled

“…The TIMSS and PISA approaches to student assessment and performance of different countries in these two assessments have been compared in a number of studies, e. g. [Olsen 2005;Grønmo, Olsen 2006;Hutchison, Schagen 2007;Tyumeneva, Valdman, Carnoy 2014;Polivanova 2015;Carnoy et al 2016;Klieme 2016]. However, most of these, with the exception of [Olsen 2005], have been based on analyzing the mathematical competencies of students in both TIMSS and PISA. The aim of TIMSS is defined as to assess knowledge and skills acquired as a result of learning mathematics and science .…”