This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.
Many countries are increasingly being challenged to integrate their growing immigrant populations. A major key to successful integration is the educational attainment of immigrant offspring. According to the results of comparative studies, second-generation immigrant students often lag behind their non-immigrant counterparts even though the host countries perform very differently with respect to the education of immigrant offspring. This study investigates how the interplay between the degrees of stratification and standardization in education systems and the degree of ethnic school segregation affects the performance gap between non-immigrant and second-generation immigrant students in member countries of the Organisation for Economic Co-operation and Development (OECD). Based on data from three waves of the OECD Programme for International Student Assessment (PISA) study (2003, 2006, and 2009), this article presents a country fixed effects approach to analyzing repeated cross-sectional data by investigating how changes in education policies and institutional contexts are associated with non-immigrant–immigrant reading performance gap. Between-school stratification was associated with lower performance of second-generation immigrants relative to native students, particularly when paired with ethnic school segregation, whereas within-school stratification (ability grouping) was associated with higher relative performance of the immigrant students. In addition, the non-native students benefited from less standardization of educational input, because performance gaps were smaller when a country’s educational resources were distributed unequally.
As the field of education has become a highly internationalised policy field in the last decade, international organisations such as the OECD play an ever more decisive role in the dissemination of knowledge, monitoring of outcomes, and research in education policy. Although the OECD lacks any binding governance instruments to put coercion on States or to provide material incentive, it has successively expanded its competences in this field. OECD advanced its status as an expert organisation in the field of education mainly by designing and conducting the international comparative PISA study. With PISA, the OECD was able to greatly influence national education systems. Basically, States were faced with external advice based on sound empirical data that challenged existing domestic policies, politics, and ideas. One prominent case for the impact of PISA is Germany. PISA was a decisive watershed in German education policy‐making. Almost instantly after the PISA results were publicised in late 2001, a comprehensive education reform agenda was put forward in Germany. The experienced reform dynamic was highly surprising because the traditional German education system and politics were characterised by deep‐rooted historical legacies, many involved stakeholders at different levels, and reform‐hampering institutions. Hence, a backlog of grand education reforms have prevailed in Germany since the 1970s. The external pressure exerted by PISA completely changed that situation.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.