Teaching Open and Reproducible Scholarship: A Critical Review of the Evidence Base for Current Pedagogical Methods and their Outcomes

Abstract

In recent years, the scientific community has called for improvements in the credibility, robustness, and reproducibility of research, characterized by higher standards of scientific evidence, increased interest in open practices, and promotion of transparency. While progress has been positive, there is a lack of consideration about how this approach can be embedded into undergraduate and postgraduate research training. Currently, the impact of integrating an open and reproducible approach into the curriculum on student outcomes is not well articulated in the literature. Therefore, in this paper, we provide the first comprehensivereview of how integrating open and reproducible scholarship into teaching and learning may impact students, using a large-scale, collaborative, team-science approach. Our review highlighted how embedding open and reproducible scholarship may impact: (1) students’ scientific literacies (i.e., students’ understanding of open research, consumption of science, and the development of transferable skills); (2) student engagement (i.e., motivation and engagement with learning, collaboration, and engagement in open research), and (3) students’attitudes towards science (i.e., trust in science and confidence in research findings). Our review also identified a need for more robust and rigorous methods within evaluations of teaching practice. We discuss implications for teaching and learning scholarship in this area.

The Stark realities of reproducible statistically orientated sociological research: Some newer rules of the sociological method

“…Specific training in undertaking transparent and reproducible social science is in its infancy. Pownall et al (2022) provide the first comprehensive review of how integrating open and reproducible scholarship into teaching and learning may impact students, using a large-scale, collaborative, team-science approach. A range of possible tools and infrastructural resources are introduced in Kitzes et al (2018), and Christensen et al (2019) is a textbook on transparent and reproducible social science research.…”

Development of a Concept Inventory on Open and Transparent Research Practices

“…These approaches have clear benefits in providing hands-on, practical training in open and transparent methods, as well as providing students meaningful opportunities to contribute to the scientific literature through replication and extension. Although these educational efforts are increasingly common, attempts to systematically assess students' conceptual understanding of open and transparent research practices have been lacking(Pownall et al, 2022). Our results indicate that the OSCI can be a useful pedagogical tool for measuring students' ability to reason about the nature of scientific evidence.…”

“…However, in most cases, these educational efforts have not been accompanied by systematic evaluation of whether they in fact improve students' conceptual foundations for conducting research (Pownall et al, 2022; see also Linn et al, 2015). After targeted instruction, researchers do not typically examine directly learning gains about open science concepts.…”

Development of the Brief Open Research Survey (BORS) to measure awareness and uptake of Open Research practices

“…Open Research training is also evident in some UKRN institutions (University of Reading, 2022; University of Surrey, 2022), largely tailored to their own institutional staff and students. However, the wider uptake of these resources across disciplines and institutions remains unclear (Pownall et al, 2022) from feminist and intersectional perspectives has been highlighted Sabik et al, 2021;Whitaker & Guest, 2020), as well as the need for increased relevance beyond quantitative methods to qualitative and mixed methods research (Branney et al, 2019;Humphreys et al, 2021).…”